This presentation is from Performance Marketing Summit 2017 (March 14, 2017 in Austin, TX). Session description: How A/B tests lie to us, and how to drive genuine improvement.

1. A/B PUZZLE
How A/B tests lie to us, and
how to drive genuine improvement
Jason Cohen, founder & CTO
@asmartbear http://wpengine.com

2. The puzzle

3. 758 out of 24,483 visits
85% confidence

4. What do stats
really tell us?

6. 758 out of 24,483 visits
85% confidence

7. Finding the Cheaters
105 athletes.
100 are clean.
5 are juicing.
Test is 95% accurate.

8. Finding the Cheaters
IN REALITY
Clean (100) Juicing (5)
TEST
RESULT
(95%
accuracy)
Clean
Juicing 5
095
5

9. Insight:
Rare events
are indistinguishable from
random noise

10. 758 out of 24,483 visits
85% confidence

11. Easy significance testing
with hamsters
Trust in the Hamster.
bit.ly/abhamster

12. Easy significance testing
N = A+B = total # of conversions.
D = (A–B)/2 = half the difference.
Significant if D2
> N.
bit.ly/abhamster
N = 758 + 685 = 1443
D = ( 758 — 685 ) / 2 = 37
1369 > 1443?
No! Not significant.

13. Final Result: As predicted
Wasted another month

14. 30
40
50
60
70
12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11
OtherInbox Splash Page

16. Trust the Hamster

17. Beware the Rare

18. Defeat rare by
seeking only huge results

19. Can you A/B test
your way to huge results?

20. 1,116
250
4,100
1,151
Individuals:
67% off
Group:
3% off

21. Wisdom of the Crowd
“I’m as clueless as you,
but in the equal and opposite direction.”

22. What’s the funniest joke?

23. And the winner is…

24. Jokes, Pizza,
Design, Features, Voice,
Values, Culture

25. Crowds find objective truth
but kill creativity

26. Beware the rare.
Seek huge results.
Don’t let data kill creativity.
Jason Cohen, founder & CTO
@asmartbear http://wpengine.com