What do your users want? Do you have a clue? Track your data and increase engagement - lessons by Netflix. Don't be short-sighted, try new experiences!

1. K E E P I N G Y O U WAT C H I N G
H O U S E O F C A R D S
B R I A N H O LT
@ H O LT B T

2. b i t . l y /
t e c h s u m m i t p r 1 6

6. W H AT D O Y O U R U S E R S WA N T ?

7. Y O U H AV E N O * * * * I N G C L U E .

8. N E I T H E R D O T H E Y.

9. S O W H Y C A N ’ T I T R A C K M Y D ATA B E F O R E A N D
A F T E R I A D D F E AT U R E S ?

10. B O O K S A L E S O N M Y W E B S I T E
0
17.5
35
52.5
70
1 2 3 4 5 6 7 8
launched new feature

12. W H AT A C T U A L LY H A P P E N E D
launched new feature
Oprah put my book in her book club
0
15
30
45
60
1 2 3 4 5 6 7 8
A B A B A B A B A B A B A B A B

13. W H AT A C T U A L LY H A P P E N E D
0
17.5
35
52.5
70
1 2 3 4 5 6 7 8
A B A B A B A B A B A B A B A B
started my a/b test
Oprah put my book in her book club

16. vs

17. 2 0 % L I F T I N V I E W I N G

18. W H Y ?

20. W E H AV E O U R T H E O R I E S B U T O F T E N I T ’ S T O U G H
T O E S TA B L I S H C A U S A L I T Y.

21. A / B T E S T I N G I S N O T A S O L U T I O N .

22. Y O U R N E W A / B T E S T

23. R E D C E L L
💩

24. 💩
B L U E C E L L
💩
💩 💩💩
💩💩💩 💩

27. G R E E N C E L L
💰

28. I F Y O U R E F I N E A 💩 E X P E R I E N C E V I A A / B
T E S T I N G , Y O U E N D U P W I T H A R E F I N E D
💩 E X P E R I E N C E

29. T RY E N T I R E LY N E W E X P E R I E N C E S T O O 💰

30. A / B T E S T S
• DO
• Verify hypothesis
• Indicate positive directions to go in
• Identify areas of consumer sensitivity
• Identify the key metrics you’re tracking (revenue versus sign-ups)
• DON’T
• Create your hypothesis after you have run the test
• Introduce your bias
• Assume that A/B testing will give you answers

31. D O N ’ T B E S H O R T- S I G H T E D
• Some tests may indicate positive short-term results but hide long-term
wins.
• e.g. SEO win versus monetization win

32. A D V I C E ?
• Simpler is almost always better
• Faster
• Less options and less choices
• Less friction
• Make choices you know most users will make; allow other users the
escape hatch*

33. A / B T E S T I N G H A R D PA R T S
• Be willing to throw work out
• Be willing to do ideas and features you don’t like
• Realize your consumers will tell you one thing but act another way
• Be willing to ignore the indication of the results

34. K E E P I N G Y O U WAT C H I N G H O U S E O F C A R D S ?
• We A/B test everything at Netflix. Every button, every layout, every
algorithm, every color. It’s all A/B tested to be the least friction possible.
• That keeps the interface intuitive and out of the way between you Frank
Underwood’s hostile Washington DC takeover.

35. H O W D O I A / B T E S T ?
• Open Source
• PlanOut (Facebook)
• SixPack (SeatGeek)
• Abba (Stripe)
• Proctor (Indeed)

36. H O W D O I A / B T E S T ?
• Commercial
• Optimizely
• Google Analytics Experiments
• KISSmetrics

37. B R I A N H O LT
@ H O LT B T