Experimentation is a crucial capability to master, yet few companies are able to. Why is this the case? Kevin Anderson, Product Manager at Vista (and previously ING), has spent years understanding the pain points that keep organizations from experimenting regularly. And he hasn’t stopped there. He has also developed a solution framework to address these issues. In this session, Kevin will not only show you why experimentation is important, but also why you need to go beyond Conversion Rate Optimization and embed experimentation in the very culture of the company, something practiced by the leaders like Vista, Booking & Airbnb. Continuous Experimentation is a core capability that your organization needs to master. This session is a great primer on how to get started.

1. Continuous
Experimentation

2. Hi, I’m Kevin Anderson
Product Manager Experimentation

4. I am right
No, I am right

5. 5

6. I believe we are the best place in the world to fail (we have plenty of practice!), and
failure and invention are inseparable twins. To invent you have to experiment,
and if you know in advance that it's going to work, it's not an experiment.
– Jeff Bezos, CEO Amazon
“

7. If you want to gain a competitive advantage, your
firm should build an experimentation capability
and master the science of conducting online tests.
– Ron Kohavi, Manager Analysis & Experimentation, Microsoft
“

8. • over 100 teams run experiments
• over 1.000 concurrent tests running
It's not about how much data we have, but about the data we need to prove
the thing that you want to prove. Experiments are a way to create data.
– Lukas Vermeer, Director of Experimentation at Booking.com
“
data
you
have
data
you
need

10. Our topic for today
source: https://towardsdatascience.com/the-experimentation-gap-3f5d374d354c

11. Making new things is more
fun
Targets on delivery, not impact
UNITE: ‘first migrate, then
upgrade
Already so many other important things to
do
There is no support group
Skills are not developed
Not possible at ING Mobile Banking
App
Not possible at ING Investment App
DA’s often not in devops
squads
UX delivers just 1 design
NUMBER OF
EXPERIMENTS
5
NO
EXPERIENCE
4 TECHNOLOGY
2 NO NEED 1 NO PRIORITY
3
NOT AWARE
OF VALUE
We don’t take the time for optimization after go live
Leadership is not asking for experiment
results
A/B testing is an option, not the default
6
Not enough conversions
NOT RELEVANT
Agile Coach are not aware
No overview of all lessons learned
No appreciation from
colleagues
Big campaigns are more visible
Few pure bizdevops squads
We do what we know best:
adhoc campaigns
Perceived as too much of a
hassle
No knowledge of exp tooling
“I know what customers want”
lack of experimentation mindset / culture
Not enough online traffic
(for some
journeys)
Development exp. platform stalls

12. My bias
+ +

13. Other relevant work

14. Expanding Experimentation
CRO
▪ Too often seen as a tactic
▪ Focused on winning tests
▪ The name is wrong
→ It limits the impact we can make
Challenges

15. credits: Craig Sullivan

16. Expanding Experimentation
CRO
Embedded
Experimentation
CRO

17. CRO vs Embedded Experimentation
Product team
Central CRO
Customer
Journey
A
B
A
B
A
B
A
B
A
B
CRO
team
Role Product team Role CRO team
▪ Build features
▪ Maintain features
▪ Create test hypothesis
▪ Create test variants
▪ Run A/B test
▪ Convince Product team to
implement winners
▪ Share learnings across org
“Do the work”
Customer
Journey
Embedded Experimentation
Product team A
B
A
B
A
B
A
B
A
B
CRO
team
vs
“Enable others”
Role Product team Role CRO team
▪ Build features
▪ Create test hypothesis
▪ Create test variants
▪ Run A/B test
▪ Maintain features
▪ Provide experimentation tools
▪ Educate teams on A/B testing
▪ Share learnings across org
vs

18. Leaders in experimentation do this
Product team
Central CRO
Customer
Journey
A
B
A
B
A
B
A
B
A
B
CRO
team
Customer
Journey
Embedded Experimentation
Product team A
B
A
B
A
B
A
B
A
B
CRO
team
vs
“Do the work”
Customer
Journey
Embedded Experimentation
Product team A
B
A
B
A
B
A
B
A
B
CRO
team
vs
“Enable others”

19. Customer Experience
Tribes
& squads
CMS
CODE
customer

20. Most CRO is ‘hacking the website’
Tribes
& squads
Risk
CMS
CODE
customer
External
developers
Client-side
JavaScript
injection

21. Integration is key
Tribes
& squads
Analytics
Personalisation
Experimentation
Assign
Pilots / Rollouts A/B tests
Dashboard
Manage Data Library
Target
Sticky ID
Toggle
Secure Trust
Statistics
Searchable
Tags
API
CMS
CODE
Server-side
Integration
customer

22. How easy is it to make a
change?
Create teams with end-to-end responsibility and provide self-service tools
1.

23. How integrated is
experimentation?
Running an experimentation should cost almost no additional
effort
2.

24. Are you willing to let go of
‘doing CRO’ and become the
Centre of Excellence (CoE)?
We need others to own experience creation
3.

25. Expanding Experimentation
CRO
Embedded
Experimentation
CRO
Getting
strategic

26. Experimentation > CRO

27. Experimentation =
Applying the scientific method
in our business

28. Scientific Method Observe
Hypothesize
Predictions
Experiments

30. credits: David Bland

31. Continuous Experimentation =
Discovery + Delivery
Continuous
Experimentation
Learn Research
Hypothesize
Prioritize
Build
based on idea from: Ton Wesseling

32. Validate strategic
programs
What are the underlying
assumptions?
How do we know these are true?
What evidence can we gather?
Should we pivot or even stop?

33. How often do you discuss strategy with
Sr. leaders?
Ask them what the big strategic (and risky) initiatives are, and help (in)validate them with
experiments
4.

34. Applying Causal Inference

35. Where should we apply more advanced
techniques?
What are the limits of a randomized controlled trial and where do we need to look for other
methods?
5.

36. HR pilot ‘unlimited leave’
Pilot group
Reference
group
(n=795)
Experiment
group
(n=303)
Control
group
(n=334)
Possibility to take ‘unlimited leave’
Normal leave schemes
Normal leave schemes

38. Which other parts of the business should
experiment?
What are their big strategic initiatives?
6.

39. Expanding Experimentation
CRO
Embedded
Experimentation
CRO
Culture of
Experimentation
Getting
strategic

40. Educate the methodology

41. That is a lot,
right?

42. 42

43. Roles of the
(near) future
Build & integrate tooling
Trainers & consultants to teach
Leaders with mandate to build this
capability
Sr. Managers asking for experiment results
Marketers, Data Analysts, UX Designers,
Developers that understand & apply
experimentation

44. What will be the next step in my
career?
How can I leverage my current knowledge and do more of what I really love doing?
7.

45. The 7 big questions
you should be
asking yourself
1. How easy is it to make a change?
2. How integrated is experimentation?
3. Are you willing to let go of ‘doing CRO’?
4. How often do you discuss strategy with
leaders?
5. Where to apply more advanced techniques?
6. Which other departments should
experiment?
7. What will be the next step in my career?

46. If you start working on answering
and solving these questions, you
will be heading towards a true
Culture of Experimentation

47. Thank you
Let’s discuss.
@kevinanderson
/kevinanderson
hi@kevinanderson.nl
Subscribe
Experimental Mind newsletter
>> kevinanderson.nl/newsletter