Learn the real best practices and pitfalls of experimentation based on scientific research and insights. Hazjier is co-author of three studies on experimentation with Harvard Business School and his work is covered in the book Experimentation Works. This talk will dive into the best practices of experiment design, the role of hierarchy in experimentation teams, and the value of experimentation.

1. 1
Experimentation is everywhere!
DESIGN THINKINGAGILE LEAN

2. 2
12%
14%
32%
26%
10%
5%
No change or unsure Increased revenues by
1-4%
Increased revenues by
5-9%
Increased revenues by
10-14%
Increased revenues by
15-19%
Increased revenues by
20%+
SOURCE: “How to Succeed in the Digital Experience Economy” (March 2019)
Three quarters of companies surveyed say experimentation
improved digital revenues by over 5%
n = 808 companies, >500 employees, March 2019

3. 3
Last year, professors at Harvard Business School and Duke Fuqua
published the largest analysis of the value of Experimentation
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018
Working Paper 20-018
Copyright © 2019 by Rembrand Koning, Sharique Hasan, and Aaron Chatterji
Experimentation and Startup
Performance: Evidence from
A/B Testing
Rembrand Koning
Harvard Business School
Sharique Hasan
Fuqua School of Business, Duke University
Aaron Chatterji
Fuqua School of Business, Duke University and
NBER
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018

4. Benefits of one year of experimentation for startups
n = 35,913 startups, 2015 – 2018
PAGEVIEWS
TIME ON SITE
FEATURES LAUNCHED
VC FUNDS RAISED
+12%
+4%
+9-18%
+10%
>99.9%
>99.0%
>99.0%
>95.0%
Significance
SOURCE: “Experimentation and Startup Performance” (Koning, Hassan, Chatterji 2019)

5. What makes for a great experiment?
What creates a great culture for experimentation?

6. 6
Internal research Academic collaborations
The Effects of Hierarchy on Learning
and Performance in Business
Experimentation
Sourobh Ghosh
Stefan Thomke
Hazjier Pourkhalkhali
Working Paper 20-081
The Effects of Hierarchy on Learning
and Performance in Business
Experimentation
Sourobh Ghosh
Stefan Thomke
Hazjier Pourkhalkhali
Working Paper 20-081
The Effects of Hierarchy on Learning
and Performance in Business
Experimentation
Sourobh Ghosh
Stefan Thomke
Hazjier Pourkhalkhali
Working Paper 20-081
The Effects of Hierarchy on Learning
and Performance in Business
Experimentation
Sourobh Ghosh
Stefan Thomke
Hazjier Pourkhalkhali
Working Paper 20-081
GLOBAL OPTIMIZATION
BENCHMARK
Insights and best
practices from over
1,000 companies and
100,000 experiments

7. 1
The Effects of Hierarchy on
Learning and Performance in Online Experimentation
Sourobh Ghosh1
, Stefan Thomke
Harvard Business School
Hazjier Pourkhalkhali
Optimizely
ABSTRACT
Do senior managers help or hurt business experiments? Despite the widespread adoption
of business experiments to guide strategic decision-making, we lack a scholarly understanding of
what role senior managers play in firm experimentation. Using proprietary data of live business
experiments from the widely-used A/B testing platform, Optimizely, this paper estimates the
influence of senior managers on learning from experiments and their performance outcomes
across industries and contexts. Our findings suggest that senior management’s impact is mixed.
On the one hand, senior managers’ involvement associates with bolder experiments that create
more statistically significant learning signals aiding in the exploration of new strategic
directions. On the other hand, their involvement may undermine parallel testing and cause-and-
effect learning that is instrumental to optimization and performance improvements. Our results
contribute to a burgeoning literature on experimentation in strategy, while helping articulate
limits that organizational design might place on data-driven decision-making. Furthermore, we
articulate different experimental learning modes in the formation of strategy, offering important
implications for how managers can modulate search and performance outcomes.
1
We gratefully acknowledge feedback from Rory McDonald, Jan Rivkin, and Andy Wu. Special thanks
to Charles Pensig and Optimizely, Inc, for making data available for this research. We acknowledge
financial support from the HBS Division of Research and Faculty Development. This draft is not
approved for public circulation. All errors and omissions in this draft are those of the authors.

8. 8
The Effects of Hierarchy on
Learning and Performance in Online Experimentation
Sourobh Ghosh1
, Stefan Thomke
Harvard Business School
Hazjier Pourkhalkhali
Optimizely
ABSTRACT
Do senior managers help or hurt business experiments? Despite the widespread adoption
of business experiments to guide strategic decision-making, we lack a scholarly understanding of
what role senior managers play in firm experimentation. Using proprietary data of live business

9. 9
Higher levels of seniority in experimentation programs associate with
SOURCE: “The Effects of Hierarchy on Learning and Performance in Business Experimentation” (Ghosh, Thomke, Pourkhalkhali 2020)
Experiments are more likely to win
Yet experiment uplifts become smaller

10. 10
Challenges from Hierarchy Benefits of Hierarchy
Overestimate ability to
influence the future
Less likely to revise opinions
in the face of conflicting data
Likely to use decision criteria
that are no longer best
practice
1 Increase freedom and confidence
of employees to take risks
Ability to balance exploitation
and exploration
Effectively accelerate adoption
of new strategies and
techniques

11. You need to ask yourself two big
questions:
How willing are you to be confronted
every day by how wrong you are?
And how much autonomy are you willing
to give to the people who work for you?
And if the answer is that you don’t like to
be proven wrong and don’t want
employees decide the future of your
products, it’s not going to work.
– David Vismans
Chief Product Officer, Booking.com
“
”

12. C D EA B
1. Number of Variations
2. Complexity:
Degree of
Change Within
Variations

13. 2 Variations
3 Variations
4 Variations
>5 Variations
77%
14%
5%
3%
Experimentsrunby numberof variations
SOURCE: Global Optimization Benchmark (Optimizely)

14. 2 Variations
3 Variations
4 Variations
>5 Variations
Significant uplift
Significant reduction
Inconclusive
77%
14%
5%
3%
SOURCE: Global Optimization Benchmark (Optimizely)

15. +75%
+48%
+32%
25%
33%
37%
44% +75%
2 Variations
3 Variations
4 Variations
>5 Variations
Significant uplift
Significant reduction
Inconclusive
Teamswiththefreedomtotestmorevariationsarefarmoresuccessful
SOURCE: Global Optimization Benchmark (Optimizely)

16. More variations associate with higher lifts and higher likelihood of detecting a
statistically significant uplifts
SOURCE:
More variants associate with more significant uplifts More variants associate with higher uplifts

17. 2.1X
Development resources are crucial to long-term success
8%
10%
11%
13%
15%
1 – 5
6 – 10
11 – 20
21 – 50
51 – 100
17%>100
Lines of Code / Variant Significant Uplift on Primary Metric

18. More complexity associates with higher lifts and higher likelihood of detecting a
statistically significant uplifts
SOURCE: Preliminary analyses
More complex tests associate with higher upliftsComplex tests associate with more significant uplifts

19. 19
Best practices by which leading performers realize more value from
experimentation
 Increase testing velocity
 Test across more channels
and properties
Scale Execution
 Manage process and reduce
bottlenecks
 Implement winners faster
 Track and publish results
frequently
Scope
 Test where the money is
 Optimize profitability /
product mix
 Experiment on cost-levers
Risk Mitigation
 De-risk product strategy
decisions through testing
 Implement feature flags to
roll back quickly if needed
Quality
 Improve research and
ideation
 Run more complex
experiments
 Include more variations per
test

20. We are constantly researching the characteristics and impact of
high performing experimentation programs
● Analysis of over 1,000 companies and more than
100,000 experiments
● Identification of best practices for experimentation
Global Optimization Benchmark
● Analysis of 14,000 experiments to identify the
best practices that make companies more
successful in experimentation
Hierarchy and Experiment Design
● Analysis of 1,000’s of experiments to understand
how risk-taking and innovation evolve over time
● Analysis of how risk-taking affects
experimentation performance
Experimentation and Innovation Experimentation and Firm Performance
● Analysis of how the scale of experimentation
affects the financial performance of organizations