Most organization that do a/b testing are just tweaking buttons or testing headlines. In this presentation you will find 5 principles of growth that can change your a/b-testing team into a real growth machine.

1. André Morys
The Million Dollar
Optimization Strategy
Uh Oh.
© Andre Morys, Web Arts AG FRANKFURT - HAMBURG - MÜNCHEN www.web-arts.com

2. André Morys
@morys
© Andre Morys, Web Arts AG 8 www.web-arts.com

3. 5.8 bn ! Retail / Lead-Volume
6
4,5
3
1,5
© Web Arts AG 9
www.web-arts.com
0
2008 2010 2012
CRO Volumen (Mrd. !)

4. 115 Conversion Experts in 15 Offices worldwide
Web Arts is founder of the
worldwide alliance of leading
conversion optimization
© Web Arts AG 10
www.web-arts.com
companies

5. Agenda today
1. Don‘t call it „Conversion Rate Optimization“
2. Optimization is not Testing.
3. Don‘t let the numbers lie to you. (Case Study)
© Andre Morys, Web Arts AG 11
www.web-arts.com

6. Part I:
It‘s about Growth, stupid.
© Andre Morys, Web Arts AG 12 www.web-arts.com

7. Who has a business plan for CRO?

8. Do You Have a Business Plan for Your CRO Strategy?
© Andre Morys, Web Arts AG 14 www.web-arts.com

9. Net Sales
© Andre Morys, Web Arts AG 15
www.web-arts.com

10. Profit Margin
Opportunity Cost:
! 500‘ Profit
© Andre Morys, Web Arts AG 16
www.web-arts.com

11. Challenge needed?
© Andre Morys, Web Arts AG 17
www.web-arts.com

12. Compare the effect of two optimization strategies:
© Andre Morys, Web Arts AG 18
www.web-arts.com

13. Net Sales
© Andre Morys, Web Arts AG 19
www.web-arts.com

14. Profit Margin
Opportunity Cost:
! 2.2m Profit
© Andre Morys, Web Arts AG 20
www.web-arts.com

15. Now you can re-invest ! 2.2m in
optimization...
© Web Arts AG 21
www.web-arts.com

16. More-Growth-Through-Testing-Rule #1
‣It‘s not about conversion rates. It‘s about
GROWTH.
‣ Set your goals properly.
‣ Measure your optimization efficiency.
‣ Average Uplift per Test
‣ Amount of Tests per Year
‣ Cumulated Uplift per Year
‣ Test Success Rate
‣ ROI per Test (%)
© Andre Morys, Web Arts AG 22
www.web-arts.com

17. „If you
double the number
of experiments
you do per year you're going to
double your
inventiveness“
Jeff Bezos, CEO amazon, 2004
© Andre Morys, Web Arts AG 23 www.web-arts.com

18. Growth of Top 10 e-Commerce Ventures in 5 Years (DE)
© Andre Morys, Web Arts AG 24
www.web-arts.com

19. The Zalando Effect?
© Web Arts AG www.web-arts.com
1200
900
600
300
0
2008 2009 2010 2011 2012
25

20. So, what‘s the problem?
© Web Arts AG 26
www.web-arts.com

21. For a 25% Compounded Global CR Increase, You Need...
Problem #3:
Most CMOs still prefer to
buy 25% more traffic.
2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 10 Tests à 2.5%?
Problem #1:
Organizations are too lame
to execute 20 tests per year.
5 5 5 5 5 5 5 5 5 5 10 Tests à 5%?
© Andre Morys, Web Arts AG 27
www.web-arts.com
5 5 5 5 5
20 Tests à 5%
and 50%
5 5 5 5 5 success rate?
Problem #2:
This is really expensive and
complex. Organizations hate
both.

22. Reality Check:
„We do 200 experiments per year. We
test everything. We don‘t need help.“
„Whats your success rate?“
„10%“
© Andre Morys, Web Arts AG 28
www.web-arts.com

23. Wait...
© Web Arts AG 29
www.web-arts.com

24. Why only 5% average uplift per test?
‣In the last case study, they talked about 45%
Uplift!
-> E-Commerce conversion rate is not email
subscription or download CR.
‣My last test had 33% Uplift!
-> Was it real? Don‘t confuse statistical
significance with validity. Tests should run
around 4 weeks for a sufficient sample.
© Andre Morys, Web Arts AG 30
www.web-arts.com

25. A/B Test Result
„I can‘t feel the uplift!“
Reality
© Andre Morys, Web Arts AG 31 www.web-arts.com

26. Testresults - Significance versus Validity
Run your experiments longer to get valid insights!
1 week
500 vs. 550 conversions
+10% Uplift
significant - wohoo!
2 weeks
1000 vs. 1030 conversions
+3% Uplift
not significant - :-(
© Andre Morys, Web Arts AG 32 www.web-arts.com

27. KPI: Average E-Commerce CR Uplift per Template Type
10 %
7,5 %
5 %
2,5 %
9,81 %
7,3 %
6,59 %
9,28 %
6,35 %
3,32 %
© Andre Morys, Web Arts AG 33
www.web-arts.com
0 %
Homepage Category Product Page Cart Checkout Site Wide
Average Uplift by Template Type (n= 134)

28. More-Growth-Through-Testing-Rule #2
‣It‘s all about effectiveness.
‣ To increase real CR, you need many really good test results.
‣ Average Uplift per Test influences your growth directly.
‣ So: Do the right things!
‣ Don‘t do it too fast.
‣ Don‘t test everything. It is expensive.
© Andre Morys, Web Arts AG 34
www.web-arts.com

29. Part II:
Where do good optimization ideas
come from?
© Andre Morys, Web Arts AG 35 www.web-arts.com

30. © Andre Morys, Web Arts AG 37 www.web-arts.com

31. where would you buy?
why?
© Andre Morys, Web Arts AG 39 www.web-arts.com

32. © Andre Morys, Web Arts AG 40 www.web-arts.com

33. quantitative qualitative
How much? Why?
© Andre Morys, Web Arts AG 41 www.web-arts.com

34. Traffic
Website
Web
Analytics
© Andre Morys, Web Arts AG 42 www.web-arts.com

35. Traffic
prefrontal Cortex
Limbic System
Perception
Website
Web
Analytics
Click
Motor Control
Hand
© Andre Morys, Web Arts AG 43 www.web-arts.com

36. Traffic
prefrontal Cortex
Limbic System
Perception
99%
Website
Web
Analytics
Click
Motor Control
Hand
1%
© Andre Morys, Web Arts AG 44 www.web-arts.com

37. Test: facebook Acount Deaktivierung
fb cancellation seite
beispiel
© Andre Morys, Web Arts AG 46
www.web-arts.com

38. Test: facebook Acount Deaktivierung
© Andre Morys, Web Arts AG 47
www.web-arts.com

39. ROI
„I have to buy it.“
Brand, Emotion, Value
„I want to buy it.“
Psychology
„I can buy it.“
Usability, Accessability,
Visibility
Web Arts ROI Pyramid

40. 7E Framework
Factors, that influence the chance of conversion, can be
identified with a conversion framework
© Web Arts AG 49 www.web-arts.com

41. © Andre Morys, Web Arts AG FRANKFURT - HAMBURG - MÜNCHEN www.web-arts.com

42. © Web Arts AG 62 www.web-arts.com

43. You are too dumb to
buy here.

45. Conversion Optimization means:
„Get a relevant good unique feeling
value
proposition in the users mind.“
© Web Arts AG 71 www.web-arts.com

46. More-Growth-Through-Testing-Rule #3
‣It is not hard to implement a testing tool. It‘s hard
to find the right thing to optimize.
‣ Find out, what motivates users.
‣ Analyse your website from different perspectives:
‣ Web Analytics: Where is the problem?
‣ User Testing: How do useres think about it?
‣ Persona: What do they not tell us?
‣ Expert Evaluation: Which psychological principles could we use?
© Andre Morys, Web Arts AG 72
www.web-arts.com

47. Conversion DMAIC®
Lean Innovation Process by Web Arts AG www.web-arts.com
DEFINE
quantitative
LEARN
qualtitative
MEASURE
© Andre Morys, Web Arts AG 73
www.web-arts.com
BACKLOG
CONTROL
IMPROVE ANALYSE

48. André Morys
CEO
Web Arts AG
andre.morys@web-arts.com
Tel. +49.6172.68097-15
www.web-arts.com
Thank You!
© Andre Morys, Web Arts AG 90 www.web-arts.com