A/B testing is a great way to really learn from user behavior, but this is mostly not the main objective of an optimization program. It needs to improve revenue! If you're only implementing significant results, you're leaving a lot of money on the table. We will explain how we run optimization at the biggest hotel chain in the Netherlands.

1. SUBTITLE BELOW
Optimize for €€€
HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

2. @AM_Klaassen / @tomvdberg1
How we got here…

3. @AM_Klaassen / @tomvdberg1
The quest for female speakers…

4. @AM_Klaassen / @tomvdberg1
Female speakers…
?

5. @AM_Klaassen / @tomvdberg1
Our lovely colleagues

6. What we do…

7. @AM_Klaassen / @tomvdberg1
Conversion rate optimization
Analytics Psychology

8. Lots of A/B-tests

9. @AM_Klaassen / @tomvdberg1
Our clients

10. @AM_Klaassen / @tomvdberg1
Adding
direct value
Learning
user behavior

11. @AM_Klaassen / @tomvdberg1

12. @AM_Klaassen / @tomvdberg1

13. @AM_Klaassen / @tomvdberg1
Van der Valk
 Unique website for each hotel
 1 centralized team
 Paid by % of their turnover

14. @AM_Klaassen / @tomvdberg1
Optimization of Valk
 Increase revenue through CRO
 Reduce costs per experiment

15. “We’re not in the business of
science, we’re in the business
of making money”

16. 3 Tactics

17. 1.
Free test software

18. @AM_Klaassen / @tomvdberg1
Jorrin Quest
GTM testing
http://gtmtesting.com

19. @AM_Klaassen / @tomvdberg1
GTM testing
 Easy setup
 Free
 Preview mode

20. 2.
Change of statistics

21. @AM_Klaassen / @tomvdberg1
Focus on finding proof

22. @AM_Klaassen / @tomvdberg1
Example test: Order flow

23. @AM_Klaassen / @tomvdberg1
Example test: Set completion
A B

24. @AM_Klaassen / @tomvdberg1
Test result

25. @AM_Klaassen / @tomvdberg1
• No effect on conversion
 Don’t implement
 Re-test with higher volumes
Conclusion

26. @AM_Klaassen / @tomvdberg1

27. @AM_Klaassen / @tomvdberg1
What’s the alternative?
Frequentist
statistics
Bayesian
statistics

28. @AM_Klaassen / @tomvdberg1
Example test: Set completion
A B

29. @AM_Klaassen / @tomvdberg1
Bayesian Test evaluation

30. @AM_Klaassen / @tomvdberg1
Focus on risk assessment
88,4%
A test result is the probability that B outperforms A:
ranging from 0% - 100%

31. @AM_Klaassen / @tomvdberg1
IMPLEMENT B PROBABILITY * EFFECT ON REVENU
Expected risk 11,6% - € 69.791
Expected uplift 88,4% € 213.530
Contribution € 180.733
* Based on 6 months and an average order value of € 175
Make a risk assessment

32. @AM_Klaassen / @tomvdberg1
Implement B
B

33. @AM_Klaassen / @tomvdberg1
abtestguide.com/bayesian/
Roy Schieving
Annemarie Klaassen

34. @AM_Klaassen / @tomvdberg1
Conclusion Bayesian
 Easier to understand and
communicate
 Better suits the business
 Don’t throw away good ideas
(indicatively significant)
 Higher implementation rate and
revenue
€ 0
€ 1.000.000
€ 2.000.000
€ 3.000.000
€ 4.000.000
€ 5.000.000
€ 6.000.000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Expected impact on revenue in 1 year
€ Frequentist € Bayesian

35. 3.
Change of test method

36. @AM_Klaassen / @tomvdberg1
50%
50%
A/B-test

37. @AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
Profit or loss?

38. @AM_Klaassen / @tomvdberg1
What’s the alternative?
A/B-test Bandits

39. If you have multiple options:
which bandit
will pay-out the
most?

40. @AM_Klaassen / @tomvdberg1
Explore Exploit
?
Explore / Exploit dilemma

41. @AM_Klaassen / @tomvdberg1
Bandit
100%

42. @AM_Klaassen / @tomvdberg1
Bandit
80%
20%
10%
90%

43. @AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
More Profit!

44. @AM_Klaassen / @tomvdberg1
Limited time frame Automation
When use Bandits?
Earn over learn

45. How?

46. @AM_Klaassen / @tomvdberg1
Example 1

47. @AM_Klaassen / @tomvdberg1
Example 1
 Older target group
 Comparing price on
other sites
 Behavior of others

48. @AM_Klaassen / @tomvdberg1
Example 1
Control No message
Variation B Copy default
Variaton C
Variation D

49. 11,1%
10,6%
10,4%
10,5%

50. @AM_Klaassen / @tomvdberg1
Example 1 – Effect on revenue
A
B
Traffic
Time
+ € 6.637

51. @AM_Klaassen / @tomvdberg1
Example 2
Control New control
Variation B Copy control
Variation C
Variation D

52. 9,2%
8,9%
7,8%
8,4%

53. @AM_Klaassen / @tomvdberg1
Example 2 – Effect on revenue
A
B
Traffic
Time
+ € 19.862

54. @AM_Klaassen / @tomvdberg1
Conclusion Bandits
 Less regret during testing
 No implementation costs
 Optimizing per user segment

55. 1. Free test software
2. Bayesian statistics
3. Bandit testing

56. revenue
Maximize

57. @AM_Klaassen / @tomvdberg1

58. “We’re not in the business of
science, we’re in the business
of making money”

59. TÄNAN VÄGA!
Bayesian calculator: abtestguide.com/bayesian/
Slide deck: ondi.me/elitecamp2016
@AM_Klaassen
annemarie@onlinedialogue.com
nl.linkedin.com/in/amklaassen
@tomvdberg1
tom@onlinedialogue.com
nl.linkedin.com/in/vandenbergtom