Individual tests drive insights & ROI, but the most sophisticated optimizers look beyond what an individual test is telling them and use data to optimize their overall testing performance. In this talk, Claire will dive into the specifics of how to track, improve, and drive insight from performance metrics for your conversion program, so you can not only run better tests, but get more out of your investment in CRO.

1. EXPERIMENT ENGINEEEClaire Vo @clairevo
CRO Metrics for
Performance & Insight
Beyond test-by-test results:
measuring testing
program effectiveness

2. Things I know to be true:
EXPERIMENT ENGINEEEClaire Vo @clairevo

3. Most CRO teams focus on
and measure the outputs of testing.
EXPERIMENT ENGINEEEClaire Vo @clairevo
(wins make us feel warm & fuzzy)

4. Claire Vo @clairevo

5. Claire Vo @clairevo

6. Claire Vo @clairevo
Justin Rondeau
Busted! DigitalMarketer Calls Bull$hit on 4 Conversion
Rate Optimization (CRO) Case Studies
http://www.digitalmarketer.com/conversion-rate-optimization-case-studies/

7. But more than anything,
the success of a testing program
is driven by the inputs.
EXPERIMENT ENGINEEEClaire Vo @clairevo

8. How to do testing, better:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Increase Tests Run (Quantity)
2. Increase Tests Won (Quality)
3. Profit

9. Claire Vo @clairevo

10. How to do testing, better—really:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Set Goals
2. Measure Performance Regularly
3. Adjust & Iterate

11. We need a set of CRO
performance metrics
to track and improve CRO
quantity and quality.
EXPERIMENT ENGINEEEClaire Vo @clairevo

12. Quantity
EXPERIMENT ENGINEEEClaire Vo @clairevo

13. EXPERIMENT ENGINEEEClaire Vo @clairevo
“We run [X] tests per week.”

14. EXPERIMENT ENGINEEEClaire Vo @clairevo
“We run [X] tests per week.”
…usually.”

15. Claire Vo @clairevo
Instead you should know:
What to know Metric to Measure
How many tests can I run Testing Capacity
How many am I running? Testing Velocity & Coverage
Am I getting any better? Trends Over Time
When I’m not running tests, why? Annotations

16. Claire Vo @clairevo
Testing Capacity
CRO PERFORMANCE METRICS
How many tests can you run per year
52
Test Duration (in weeks)
x # of simultaneously
testable pages/funnels

17. Claire Vo @clairevo
Testing Velocity
CRO PERFORMANCE METRICS
How many experiments are run per [time period]
0
2.5
5
7.5
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
• Weekly for high traffic sites
• Monthly for low traffic sitesGoal 8 tests / week

18. Claire Vo @clairevo
Testing Velocity
CRO PERFORMANCE METRICS
How many experiments are run per [time period]
-0.4
2.2
4.8
7.4
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
• Weekly for high traffic sites
• Monthly for low traffic sitesGoal 8 / week
Tracking trend towards goal
BONUS

19. EXPERIMENT ENGINEEEClaire Vo @clairevo
Make sure you’re not
wasting traffic.
(use it or lose it.)

20. Claire Vo @clairevo
Łukasz Twardowski
A/B Testing and the Infinite Monkey Theory
http://www.slideshare.net/useitbetter/ab-testing-and-the-infinite-monkey-theory

21. 13%
87%
Claire Vo @clairevo
Testing Coverage
CRO PERFORMANCE METRICS
What % of testable days are you running a test?
July
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
August
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
83%

22. Claire Vo @clairevo
Time Since Last
Zero Test Day
CRO PERFORMANCE METRICS
It has been
100
DAYS SINCE WE HAD
NO TESTS LIVE

23. Quality
EXPERIMENT ENGINEEEClaire Vo @clairevo

24. EXPERIMENT ENGINEEEClaire Vo @clairevo
Stop evaluating the
quality of your program
on a test-by-test basis.
(it makes you look good, until you look bad)

25. Claire Vo @clairevo
You should also track:
What to know Metric to Measure
Am I running effective tests?
Win Rate, Lift Amount,
Expected Value
Am I running tests effectively? ROI
Am I getting any better? Trends Over Time

26. 53%
29%
19%
Claire Vo @clairevo
Win Rate
CRO PERFORMANCE METRICS
What % of tests run win / lose / are inconclusive?
13%

27. Claire Vo @clairevo
Rolling Win Rate
CRO PERFORMANCE METRICS
What % of tests run win / lose / are inconclusive over time?
0
4
8
12
16
April May June July August September October November
Goal 15% Win Rate
Tracking by Page
BONUS
Once you hit a steady testing
velocity, ideally your win rate
stays level or increases.

28. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
$1 per roll of the die
Every time you roll a 3
I pay you $5
Win Rate = 1/6 = 16%
Value of Win = $5
Expected Value of Roll = 83 cents
DO NOT PLAY THIS GAME!

29. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
On average, what is the expected value of any test run?
Win Rate
x
Average Lift of Winning Test
x
Revenue Value of Test

30. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
$1500 every time you run a test
Every time you win, you get 10% increase
on your revenue, which is $1,000,000
and you win 10% of the time
Win Rate = 10%
Value of Win = 10% x $1,000,000 = $100,000
Expected Value of Test = $10,000
RUN THIS TEST FOR SURE!

31. Claire Vo @clairevo
Annual Expected Value of Program
CRO PERFORMANCE METRICS
Expected Value of Test
x
Annual Testing Capacity
What is the estimated potential of my testing program?

32. Claire Vo @clairevo
Annual Expected Value of Program
CRO PERFORMANCE METRICS
What is the estimated potential of my testing program?
High Complexity
Low Velocity
Low Complexity
High Velocity
Type of Tests Take longer, bigger changes Easier, smaller changes
Win Rate 20% 10%
Avg Lift 30% 10%
Expected Value of Test $30,000 $10,000
Tests / Year 25 100
Annual Expected Value
of Testing Program $1.5M $1M

33. EXPERIMENT ENGINEEEClaire Vo @clairevo
Yes, you should
trend this over time.
(and set a goal!)

34. Putting it all together
EXPERIMENT ENGINEEEClaire Vo @clairevo

35. CRO Metrics for Performance:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Track quantity by measuring velocity & coverage
2. Track quality by win rate, avg. lift, and expected value
3. Set goals & measure everything over time

36. EXPERIMENT ENGINEEEClaire Vo @clairevo
Questions?
claire@experimentengine.com
experimentengine.com/cxl-live-2016