Everything you need to know to perform a kickass A/B test – start testing today!

1. The Art of A/B Testing
@mike_greenfield
numeratechoir.com
500 Startups, 2013-05-08

2. What’s an A/B Test?
Definition: An A/B Test is a means by which
a product’s users are randomly given
one of two or more experiences.
Usage: Companies use A/B Tests to
discover which experience is most
effective.

3. A/B Testing Overview
Why You Should A/B Test
What You Should A/B Test
When You Should A/B Test
How You Should A/B Test
What I’ve Learned from A/B Testing

4. Why You Should A/B Test

5. Humans’ quantitative intuition is poor
Beliefs:
1) “I have great product intuition”
2) “This is business, not science”
3) “It leads to a local maximum”
Reality:
At Internet scale, testing and measuring
properly has a huge return on investment.

6. “I have great product intuition”
• Improved signup flow: often a disaster
• Sharing on Facebook: can be good or
bad
• Home page changes: usually a mixed
bag
…but product changes != progress

7. “This is business, not science”
• Requires oversight and sleuthing
• Only works for big improvements
• Requires other factors to stay the same
…but change-then-measure is flawed

8. Test Test Type Goal
Change Button
Color
Optimization Increased clickthrough
Old Site Design
vs. New Site
Design
Holistic (aka don’t
shoot yourself in
the foot)
Make an informed
decision on old vs. new
“It leads to a local maximum”
…so test holistically

9. What You Should A/B Test

10. Test an almost viral flow
1000
signups
300
inviting
friends
3000
invitations
1500
invitation
clicks
900
signups
K-factor = 0.9
☑ A/B Test this flow!

11. Don’t test a non-viral flow
1000
signups
100
inviting
friends
600
invitations
200
invitation
clicks
80
signups
K-factor = 0.08
Spend time elsewhere

12. Test email
subject lines
Subject: The Craziest Thing
My Child Has Done
Subject: 5 Embarrassing Kid
Moments
vs.

13. Test email
subject lines
Subject: The Craziest Thing
My Child Has Done
Subject: 5 Embarrassing Kid
Moments
vs.
+153%

14. Improve purchase funnel
Get 2011 Picks
Get Bracketbrains
vs.

15. Improve purchase funnel
Get 2011 Picks
Get Bracketbrains
vs.
+30%

16. Validate your new design*
* if you have scale
Scale What to Do Reasoning
Early stage Just change it. Nothing to lose, no data to
test.
Something to lose A/B test it. The existing product is
probably more effective
than you think.
Understand the
consequences of an
“upgrade.”

17. When You Should A/B Test

18. Test only if you can get to
statistical significance
(Google: “split test calculator”)
Big: 200,000
emails with a 3%
CTR option and a
4% CTR option
-------------------------
6000 vs. 8000
Small: 200 emails
with a 3% CTR
option and a 4%
CTR option
--------------------------
6 vs. 8

19. Test only if you can get to business
significance
• Only things that can cumulatively
have a meaningful impact on your
business
• For emails, a small list means small
improvements: don’t test
• For virality, small changes matter if and
only if you’re close to K=1

20. Rule of thumb: test every user-
facing change that will be seen
by 10,000-100,000 people

21. How You Should A/B Test

22. Okay Choice: Use Commercial
Tools
• MixPanel
• Unbounce
• Optimizely
• Google Analytics

23. Best Choice: Build Your Own
Framework
• Yep, it’s work with no immediate
payoff
• Your mom won’t care
• Your users won’t care
But…
• There are simple ways to get started
• It gives you tons of flexibility

24. Why Build Your Own
• Incorporate tests in many places
(page ordering, new designs, email
content, email strategy, mobile)
• Look at results holistically
• Go back and see how any test
influences anything, not just the stats
you’re tracking

25. Code
It needs to be super simple to create a test
{ab_test_if test=“signup_reason”
option=“awesome” user=$viewer}
because it’s awesome!
{/ab_test_if}
{ab_test_if test=“signup_reason” option=“free”
user=$viewer}
because it’s free!
{/ab_test_if}

26. Data Structure
AB_TESTS (id, name, time_created)
AB_TEST_OPTIONS
(id, ab_test_id, weight, name)
USER_AB_TEST_OPTIONS
(id, user_id/visitor_id, ab_test_option_id, t
ime_created)

27. Reporting
select
AB_TEST_OPTION_ID,
ACTIVITY_ID,
count(distinct USER_ID) USERS_DOING_ACTIVITY
from USER_ACTIVITY a, USER_AB_TEST_OPTIONS b
where a.USER_ID=b.USER_ID
and AB_TEST_OPTION_ID in (…)
and ACTIVITY_ID in (…)
and a.time_created>b.time_created
GROUP BY AB_TEST_OPTION_ID, ACTIVITY_ID;

28. Scaling
• An A/B testing system can yield a lot of
DB writes
• Reporting means many long-running
SQL queries
• Need to batch several aspects

29. Run Some A/A Tests
• Test two versions of the same thing
• If results are wildly off, something’s
wrong with the testing system
• Deciding too early is a major issue: it’s
usually best to be conservative before
choosing a winner

30. More framework implementation
details: bit.ly/artofabtesting

31. What I’ve learned from A/B testing

32. Focus on 1 item in emails
• Clear subject focused on that item (Why
the Giants will win the World Series)
• Body of text focused on that item
(peripheral content is okay on the
periphery)
• Clear, big clickthrough action in the
email body (See why the Giants will win)
• Require clickthrough to get the full story

33. In signup, minimize distraction
• Provide context/messaging of what
the product is, but don’t make it
clickable
• Clear “next” or “continue” steps to
guide user through the process
• Remove unnecessary navigation

34. Highlight friends, not your product
• Most effective: your friends are doing
something; you should join them
• Unless you’re Apple, no one cares
about your new feature or new design
• People probably don’t care about
your fancy new social network
• “Join my circle because I trust you”
beats “check out this great product”

35. When possible,
TELL USERS WHAT TO DO

36. 3 Things to Remember

37. A/B testing = good culture
• Data trump opinions
• Iterate quickly but intelligently
• Everyone gets better at predicting
product success

38. Test changes if they’re likely to
have both statistical and business
significance

39. Validate the Big Stuff
A/B Test Holistically; testing
isn’t a substitute for product
vision.
Optimize the Small Stuff
The details matter more than you
think.

40. Thanks.
mike@mikegreenfield.com
@mike_greenfield