This is an updated version of the talk I gave at Product Camp ATL 2017. Atlanta doesn't talk enough about data analytics and A/B testing. I was able to follow my product's performance on a product scorecard built on (free) analytics tools to identify key areas of improvement. Once identified, I conducted A/B testing on potential fixes to find the best performing change that resulted in real revenue impact. This session is a case study on what I did and how you can do the same for your product. Note: I presented this session last year and it helped someone get a promotion at work! So, I'll be presenting an updated version of it in 2017.

1. Million Dollar
Impact Through
Metrics, Analytics, &
A/B Testing

2. Hello!
I am Azhar Bande-Ali
Software Engineer
turned PM with a
passion for people and
data
You can find me at:
@AzharBA

3. In this talk
◎ Discuss web products at scale
◎ Narrow approach due to time constraints
◎ Principles can be adapted to your world
Feel free to ask questions

9. Metrics
Define your measurement criteria

10. Top Down
Metrics
Measure what your
business cares about.
Everything else is
vanity.

11. Place your screenshot here
E-commerce Website
Improve yearly revenue from online sales

12. Metrics for growing revenue

13. Conversion Rate
Metrics for growing revenue
Drop-off Rate Return Rate

14. Conversion Rate
% of unique sessions
with a submitted
order
Metrics for growing revenue
Drop-off Rate
For every page in the
ordering flow: % of
sessions that don’t
graduate to next
step
Return Rate
% of new users who
repeat order
Compare each metric to your target/goal and industry average!

15. Analytics
Measure your metrics

16. OR

17. Getting started with Analytics
◎ Needs code change - development cost
◎ Measure:
○ Conversion rate of funnels and
○ Drop off rates on pages
◎ Event tracking for more intricate analysis

18. Marry analytics to business goals
Conversion Rate
Drop off Rate
Return Rate

19. Measure
Frameworks
◎ Funnels
◎ AIDA
○ Awareness
○ Interest
○ Desire
○ Action

20. Measure
Frameworks
◎ Funnels
◎ AIDA
○ Attention
○ Interest
○ Desire
○ Action
Events

21. Drop-offs

22. ROI

23. How to get buy in from your boss and partners

24. $1,085,136
How to get buy in from your boss and partners

25. A/B Test
Validate your hypothesis

27. Hypothesis
Making “Continue” button
more prominent will improve
the checkout conversion rate.

28. Variant 1
Remove “Cancel Order”
Button

29. Variant 2
Remove “Cancel Order”
button and rename
“Continue” to “Checkout”

30. And the winner is..

31. And the winner is..
..neither!

32. Truth about A/B testing
Diminishing Return
Even a successful A/B test variant
will eventually start failing for a
variety of reasons.
No Winners
It is possible that none of your
variants cause a big enough
change to be declared winner
Success Rate
According to KissMetrics, only 1 out
of 8 A/B tests have a valid winner
Significance
Sample being tested on should be
big enough to represent
population
Control Group
Always show a ‘no change’ state to
a control group to compare
understand what would happen if
you did nothing
Representation
The test should be conducted on
the same group of users that are a
sample of the actual user set

33. Next
Steps
Org Support
Ensure that your organization
recognizes the value of the
opportunity enough to prioritize it
Iterate
Be ready to come up with several
hypothesis about which features
create value for users
Failure
Even the best teams have a 12.5%
chance of success. Know that it
isn’t easy and you’ll fail a lot

35. Thanks!
Any questions?
You can find me at:
@AzharBA
pcampatl@azharb.com