At Microsoft, we run over a thousand online controlled experiments (A/B tests being the simplest) every month. A key goal for the Experimentation Platform is to provide scorecards you can trust. We will share real examples of surprising results and pitfalls, so you can generate trustworthy analyses.

1. Ronny Kohavi, Distinguished Engineer, General Manager,
Analysis and Experimentation, Microsoft
Joint work with Thomas Crook, Brian Frasca, and Roger Longbotham, A&E Team
A/B Testing Pitfalls
Slides at http://bit.ly/ABPitfalls

2. Ronny Kohavi 2
A/B Tests in One Slide
ØConcept is trivial
§Randomly split traffic between two (or more) versions
oA (Control)
oB (Treatment)
§Collect metrics of interest
§Analyze
ØA/B test is the simplest controlled experiment
§A/B/n refers to multiple treatments (often used and encouraged: try control + two or three treatments)
§MVT refers to multivariable designs (rarely used by our teams)
ØMust run statistical tests to confirm differences are not due to chance
ØBest scientific way to prove causality, i.e., the changes in metrics are caused by changes
introduced in the treatment(s)

3. ConversionXL Audience Statistics
Ronny Kohavi 3
13%
30%
20% 20%
17%
0%
5%
10%
15%
20%
25%
30%
35%
None Few 12 (one per
month)
15-30 Lots and lots
Experiments per year run by attendees
based on pre-conference survey (N=118)
83% of attendees ran less than 30 experiments last year.
Experimenters at Microsoft use our ExP platform to start ~30 experiments per day

4. Experimentation at Scale
ØI’ve been fortunate to work at an organization that values being data-driven (video)
ØWe finish about ~300 experiment treatments per week, mostly on Bing, MSN, but also on
Office, OneNote, Xbox, Cortana, Skype, Exchange, OneDrive.
(These are “real” useful treatments, not 3x10x10 MVT = 300.)
ØEach variant is exposed to between 100K and millions of users, sometimes tens of millions
ØAt Bing, 90% of eligible users are in experiments (10% are a global holdout changed once a year)
ØThere is no single Bing. Since a user
is exposed to over 15 concurrent
experiments, they get one of
5^15 = 30 billion variants
(debugging takes a new meaning).
ØUntil 2014, the system was
limiting usage as it scaled.
Now the limits come from
engineers’ ability to code new ideas
Ronny Kohavi 4

5. Two Valuable Real Experiments
ØWhat is a valuable experiment?
§Absolute value of delta between expected outcome and actual outcome is large
§If you thought something is going to win and it wins, you have not learned much
§If you thought it was going to win and it loses, it’s valuable (learning)
§If you thought it was “meh” and it was a breakthrough, it’s HIGHLY valuable
See http://bit.ly/expRulesOfThumb for some examples of breakthroughs
ØExperiments ran at Microsoft’s Bing with millions of users in each
ØFor each experiment, we provide the OEC, the Overall Evaluation Criterion
ØCan you guess the winner correctly? Three choices are:
oA wins (the difference is statistically significant)
oFlat: A and B are approximately the same (no stat sig diff)
oB wins
5

6. Example : Bing Ads with Site Links
ØShould Bing add “site links” to ads, which allow advertisers to offer several
destinations on ads?
ØOEC: Revenue, ads constraint to same vertical pixels on avg
ØPro adding: richer ads, users better informed where they land
ØCons: Constraint means on average 4 “A” ads vs. 3 “B” ads
Variant B is 5msc slower (compute + higher page weight)
Ronny Kohavi 6
A B
• Raise your left hand if you think A Wins (left)
• Raise your right hand if you think B Wins (right)
• Don’t raise your hand if they are the about the same

7. Bing Ads with Site Links
ØIf you raised your left hand, you were wrong
ØIf you did not raise a hand, you were wrong
ØSite links generate incremental revenue on the order of tens of millions of dollars
annually for Bing
ØThe above change was costly to implement. We made two small changes to Bing, which
took days to develop, each increased annual revenues by over $100 million
Ronny Kohavi 7

8. Example: Underlining Links
ØDoes underlining increase or decrease clickthrough-rate?
Ronny Kohavi 8

9. Example 4: Underlining Links
ØDoes underlining increase or decrease clickthrough-rate?
ØOEC: Clickthrough Rate on search engine result page (SERP) for a query
Ronny Kohavi 9
A (with underlines) B (no underlines)
• Raise your left hand if you think A Wins (left, with underlines)
• Raise your right hand if you think B Wins (right, without underlines)
• Don’t raise your hand if they are the about the same

10. Underlines
ØIf you raised your right hand, you were wrong
ØIf you did not raise a hand, you were wrong
ØUnderlines improve clickthrough-rate for both algorithmic results and ads (so more
revenue) and improve time to successful click
ØModern web designs do away with underlines, and most sites have adopted this design,
despite data showing that users click less and take more time to click
ØFor search engines (Google, Bing Yahoo), this is a very questionable industry direction
Ronny Kohavi 10

11. Pitfall 1: Misinterpreting P-values
ØNHST = Null Hypothesis Statistical Testing, the “standard” model commonly used
ØP-value <= 0.05 is the “standard” for rejecting the Null hypothesis
ØP-value is often mis-interpreted.
Here are some incorrect statements from Steve Goodman’s A Dirty Dozen
1. If P = .05, the null hypothesis has only a 5% chance of being true
2. A non-significant difference (e.g., P >.05) means there is no difference between groups
3. P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis
4. P = .05 means that if you reject the null hyp, the probability of a type I error (false positive) is only 5%
ØThe problem is that p-value gives us Prob (X >= x | H_0), whereas what we want is
Prob (H_0 | X = x)
Ronny Kohavi 11

12. Pitfall 2: Expecting Breakthroughs
Ø Breakthroughs are rare after initial optimizations.
§At Bing (well optimized), 80% of ideas fail to show value
§At other products across Microsoft, about 2/3 of ideas fail
ØTake Sessions/User, a key metric at Bing.
Historically, it improves 0.02% of the time: that’s one in 5,000 treatments we try!
ØMost of the time, we invoke Twyman’s law (http://bit.ly/twymanLaw)
ØNote relationship to prior pitfall
§With standard p-value computations, 5% of experiments will show stat-sig movement to Sessions/User
when there is no real movement (i.e., if the Null Hypothesis is true), half of those positive
§99.6% of the time, a stat-sig movement with p-value = 0.05 will be a false positive
Ronny Kohavi 12
Any figure that looks interesting or different is usually wrong

13. Pitfall 3: Not Checking for SRM
ØSRM = Sample Ratio Mismatch
ØIf you run an experiment with equal percentages assigned to Control/Treatment (A/B),
you should have approximately the same number of users in each
ØReal example from an experiment alert I received this week:
§Control: 821,588 users, Treatment: 815,482 users
§Ratio: 50.2% (should have been 50%)
§Should I be worried?
ØAbsolutely
§The p-value is 1.8e-6, so the probability of this split (or more extreme) happening by chance is less than 1
in 500,000
§Note that the above statement is not a violation of the pitfall #1 because by the experiment design, there
should be an equal number of users in control/treatment, so we want the conditional probability
P(actual split=50.2% | designed split=50%)
Ronny Kohavi 13

14. Pitfall 4: Wrong Success Metric (OEC)
ØOffice Online tested new design for homepage
ØObjective: increase sales of Office products
ØOverall Evaluation Criterion (OEC) was clicks to the Buy Button [shown in red boxes]
Which one was better?
Control
Treatment

15. Pitfall: Wrong OEC
ØTreatment had a drop in the OEC (clicks on buy) of 64%!
ØNot having the price shown in the Control lead more people to click to determine the price
ØLesson: measure what you really need to measure: actual sales
(it is more difficult at times)
ØLesson 2: Focus on long-term customer lifetime value
ØPeep in keynote here said (he was OK with me mentioning this):
§What’s the goal? More money right now
§Common pitfall: You want to optimize long-termmoney. NOT right now.
Raising prices gets you short-term money, but long-term abandonment
ØComing up with a good OEC using short-term metrics is REALLY hard

16. Example: OEC for Search
ØKDD 2012 Paper: Trustworthy Online Controlled Experiments:
Five Puzzling Outcomes Explained
ØSearch engines (Bing, Google) are evaluated on query share (distinct queries) and
revenue as long-term goals
ØPuzzle
§A ranking bug in an experiment resulted in very poor search results
§Degraded (algorithmic) search results cause users to search more to complete
their task, and ads appear more relevant
§Distinct queries went up over 10%, and revenue went up over 30%
ØThis problem is now in the book data science interviews exposed
ØWhat metrics should be in the OEC for a search engine?
Ronny Kohavi 16

17. Puzzle Explained
ØAnalyzing queries per month, we have
𝑄𝑢𝑒𝑟𝑖𝑒𝑠
𝑀𝑜𝑛𝑡ℎ
=
𝑄𝑢𝑒𝑟𝑖𝑒𝑠
𝑆𝑒𝑠𝑠𝑖𝑜𝑛
×
𝑆𝑒𝑠𝑠𝑖𝑜𝑛𝑠
𝑈𝑠𝑒𝑟
×
𝑈𝑠𝑒𝑟𝑠
𝑀𝑜𝑛𝑡ℎ
where a session begins with a query and ends with 30-minutes of inactivity.
(Ideally, we would look at tasks, not sessions).
ØKey observation: we want users to find answers and complete tasks quickly,
so queries/session should be smaller
ØIn a controlled experiment, the variants get (approximately) the same
number of users by design, so the last term is about equal
ØThe OEC should therefore include the middle term: sessions/user
Ronny Kohavi 17

18. Bad OEC Example
ØYour data scientists makes an observation:
2% of queries end up with “No results.”
ØManager: must reduce.
Assigns a team to minimize “no results” metric
ØMetric improves, but results for query
brochure paper
are crap (or in this case, paper to clean crap)
ØSometimes it *is* better to show “No Results.”
Real example from my Amazon Prime now search 3/26/2016
https://twitter.com/ronnyk/status/713949552823263234
Ronny Kohavi 18

19. Pitfall 5: Combining Data when Treatment Percent
Varies with time
ØSimplified example: 1,000,000 users per day
ØFor each individual day the Treatment is much better
ØHowever, cumulative result for Treatment is worse (Simpson’s paradox)
Conversion Rate
for two days
Friday Saturday
Total
C/T split: 99/1 C/T split: 50/50
Control
20,000
= 2.02%
5,000
= 1.00%
25,000
= 1.68%
990,000 500,000 1,490,000
Treatment
230
= 2.30%
6,000
= 1.20%
6,230
= 1.22%
10,000 500,000 510,000

20. Pitfall 6: Get the Stats Right
ØTwo very good books on A/B testing (A/B Testing from Optimizely founders
Dan Siroker and Peter Koomen; and You Should Test That by WiderFunnel’s
CEO Chris Goward) get the stats wrong (see Amazon reviews).
ØOptimizely recently updated their stats in the product to correct for this
ØBest techniques to find issues: run A/A tests
§Like an A/B test, but both variants are exactly the same
§Are users split according to the planned percentages?
§Is the data collected matching the system of record?
§Are the results showing non-significant results 95% of the time?
Ronny Kohavi 20

21. More Pitfalls
ØSee KDD paper: Seven Pitfalls to Avoid when Running Controlled Experiments on the Web
(http://bit.ly/expPitfalls)
ØIncorrectly computing confidence intervals for percent change
ØUsing standard statistical formulas for computations of variance and power
ØNeglecting to filter robots/bots
Lucrative business, as shown in photo I took ->
ØInstrumentation issues
Ronny Kohavi 21

22. The HiPPO
ØHiPPO = Highest Paid Person’s Opinion
ØWe made thousands toy HiPPOs and handed them
at Microsoft to help change the culture
ØGrab one here at ConversionXL
ØChange the culture at your company
ØFact: Hippos kill more humans than any other (non-human) mammal
ØListen to the customers and don’t let the HiPPO kill good ideas
Ronny Kohavi 22

23. Ronny Kohavi 23
Getting numbers is easy;
getting numbers you can trust is hard
Slides at http://bit.ly/ABPitfalls
See http://exp-platform.com for papers.
Plane reading booklets with selected papers available outside room
Remember this