There are times when rules can be broken. The severity of breaking or bending these rules changes as your industry and risk change. This presentation is about the shortcuts that business experimentation programs can take that still lead to business value but bend the rules of the standard scientific method. There are certain liberties that business testing programs can take that lead to a similar/the same outcome.

1. Tim Mehta
Root Insurance
The Flexibility of
Business
Experimentation

2. Experimentation Manager
● Build and optimize
experimentation/CRO programs
● Background in UX and digital
strategy
Tim Mehta
The Flexibility of Business Experimentation

3. Experimentation Manager
● Build and optimize
experimentation/CRO programs
● Background in UX and digital
strategy
Tim Mehta
The Flexibility of Business Experimentation

4. Why businesses have more flexibility
when it comes to running
experiments
+ Why?
+ Where?
+ How?
The Flexibility of Business Experimentation

5. Lower Cost
Lower Risk
Compared to:
● Pharmaceutical
● Scientific
● Psychology
● Industrial
Why businesses have more flexibility
5

6. Where businesses can leverage their
flexibility within the experimentation
lifecycles
+ Why?
+ Where?
+ How?
The Flexibility of Business Experimentation

7. Where
can we be
flexible?
1. Test measurement
2. Design or function fidelity
3. Implementation standards
4. Optimization vs. Learning

8. How businesses can bend the rules
of experimentation and still find
success
+ Why?
+ Where?
+ How?
The Flexibility of Business Experimentation

9. How businesses can be more flexible with experimentation
1. Test measurement
9

10. You don’t have to
use traditional
frequentist
measurement every
time
How businesses can be flexible: Test measurement
10
frequentist
��

11. “We need to know the level of the impact with
high confidence”
● Bayesian
● Bootstrap
11
You don’t have to
use traditional
frequentist
measurement every
time
frequentist
��
How businesses can be flexible: Test measurement

12. “We need to know the level of the impact with
high confidence”
● Bayesian
● Bootstrap
“We don’t have the traffic or sample size we need
to run tests within a reasonable time frame”
● Sequential testing
● CUPED
● Outlier capping
● Post-Stratification
● Machine learning
● Bayesian
● Bootstrap
12
You don’t have to
use traditional
frequentist
measurement every
time
frequentist
��
How businesses can be flexible: Test measurement

13. “We need to know the level of the impact with
high confidence”
● Bayesian
● Bootstrap
“We don’t have the traffic or sample size we need
to run tests within a reasonable time frame”
● Sequential testing
● CUPED
● Outlier capping
● Post-Stratification
● Machine learning
● Bayesian
● Bootstrap
13
You don’t have to
use traditional
frequentist
measurement every
time
frequentist
��
How businesses can be flexible: Test measurement

14. “We need to know the level of the impact with
high confidence”
● Bayesian
● Bootstrap
“We don’t have the traffic or sample size we need
to run tests within a reasonable time frame”
● Sequential testing
● CUPED
● Outlier capping
● Post-Stratification
● Machine learning
● Bayesian
● Bootstrap
“We only care if treatments are better, we have
no interest in deploying if they are flat or worse”
● One-tailed z-test
● Auto-shutdown / Guardrail metrics 14
You don’t have to
use traditional
frequentist
measurement every
time
frequentist
��
How businesses can be flexible: Test measurement

15. You can build
conditions &
guardrails into
Primary KPIs with
secondary metrics
Examples:
● Exclude purchases over $10k
● Exclude purchases of accessories
● Exclude purchases that ended up in a return
● Only include orders that lead to a net positive
LTV
15
🛠
How businesses can be flexible: Test measurement

16. How businesses can be more flexible with experimentation
2. Test fidelity
16

17. Your initial tests can
be very low fidelity
How businesses can be flexible: Test fidelity
17

18. Your initial tests can
be very low fidelity
18
How businesses can be flexible: Test fidelity

19. Your initial tests can
be very low fidelity
19
How businesses can be flexible: Test fidelity

20. 20
Your initial tests can
be very low fidelity
“We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
How businesses can be flexible: Test fidelity

21. 21
Your initial tests can
be very low fidelity
“We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
How businesses can be flexible: Test fidelity

22. 22
Your initial tests can
be very low fidelity
“We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
How businesses can be flexible: Test fidelity

23. “We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
“We need to make a decision on the alignment
of a business strategy”
● Use high-trafficked areas for inferential
data
23
Your initial tests can
be very low fidelity
How businesses can be flexible: Test fidelity

24. “We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
“We need to make a decision on the alignment
of a business strategy”
● Use high-trafficked areas for inferential
data
24
Your initial tests can
be very low fidelity
How businesses can be flexible: Test fidelity

25. “We are thinking of about introducing a new
feature or product”
● Painted-door experiment
● Minimum Viable Test
“We need to make a decision on the alignment
of a business strategy”
● Use high-trafficked areas for inferential
data
“We have a winning variant from this low-fidelity
experiment”
● Avoid the temptation
● Iterate into higher fidelity
25
Your initial tests can
be very low fidelity
How businesses can be flexible: Test fidelity

26. How businesses can be more flexible with experimentation
3. Implementation Standards
26

27. You can implement
non-winning tests
How businesses can be flexible: Implementation
27

28. 28
You can implement
non-winning tests
“We want to align new branding and creative
across all channels and assets”
“We are Introducing a new feature that has
long-term value and commitment”
“It’s important to get as many of these new
products off the shelf, even at the cost of overall
revenue”
How businesses can be flexible: Implementation

29. 29
“We want to align new branding and creative
across all channels and assets”
“We are introducing a new feature that has
long-term value and commitment”
“It’s important to get as many of these new
products off the shelf, even at the cost of overall
revenue”
You can implement
non-winning tests
How businesses can be flexible: Implementation

30. How businesses can be more flexible with experimentation
4. Optimization vs. Learning
30

31. You can freely
choose between
optimizing &
learning
How businesses can be flexible: Optimize vs. Learn
31
⚖
Optimization Theory

32. You can freely
choose between
optimizing &
learning
32
⚖
“We aren’t interested in learnings, we just want
to grow fast”
● Focus on execution and efficiencies
● Automate wherever possible
● Multi-armed Bandit tests
● Machine learning models
How businesses can be flexible: Optimize vs. Learn

33. You can freely
choose between
optimizing &
learning
33
⚖
“We aren’t interested in learnings, we just want
to grow fast”
● Focus on execution and efficiencies
● Automate wherever possible
● Multi-armed Bandit tests
● Machine learning models
“We want to know the reason for all behavioral
changes we detect”
● Focus on research and documentation
● Perform pre and post-test user research
● Build institutional memory
● Socialize and advocate program
How businesses can be flexible: Optimize vs. Learn

34. Where
can we be
flexible?
1. Test measurement
2. Design or function fidelity
3. Implementation standards
4. Optimization vs. Learning

35. Confidential | Do Not Distribute 35
35
Tim Mehta
Manager, Conversion Rate
Optimization @ Root Insurance
Questions?