A/B Testing Data-Driven Algorithms in the Cloud - Webinar
•
2 likes•446 views
We present how A/B testing can be used to evaluate the performance of Machine Learning algorithms. We explore the different evaluation approaches - from offline evaluation to online evaluation - with a particular focus on long-term KPIs and on the recent Cloud-based technologies that can facilitate the development and integration of A/B testing.
Recommended
More Related Content
Viewers also liked
Viewers also liked (9)
Similar to A/B Testing Data-Driven Algorithms in the Cloud - Webinar
Similar to A/B Testing Data-Driven Algorithms in the Cloud - Webinar (20)
Recently uploaded
Recently uploaded (18)
A/B Testing Data-Driven Algorithms in the Cloud - Webinar
- 1. A/B Tes(ng Data-Driven Algorithms in the Cloud cloudacademy.com 7/25/2016
- 2. About us Roberto Turrin Luca Baroffio Sr. Data Scien8st (PhD) Data Scien8st (PhD) @robytur @lucabaroffio
- 3. Agenda Data-driven algorithms Evalua8on A/B tes8ng Challenges in A/B tes8ng data-driven algorithms A/B tes8ng in the cloud Data-driven A/B tes8ng Conclusions Q&A
- 4. Data-driven algorithms Decision problems that can be modeled from data
- 5. Data-driven problems - I Image recogni8on Document classifica8on Speech-to-text Spam/fraud detec8on Stock price predic8on Content personaliza8on Market basket Search sugges8on Playlist genera8on Document clustering User segmenta8on Target Adver8sing
- 6. Data-driven problems - II Image recogni8on Document classifica8on Speech-to-text Spam/fraud detec8on Stock price predic8on Content personaliza8on Market basket Search sugges8on Playlist genera8on Document clustering User segmenta8on Target Adver8sing classifica'on regression clustering rule extrac'on ? 170 cm group A group B A, B C Supervised Unsupervised
- 7. Data-driven algorithm pipeline Training Predic6on batch real-8me Feature extrac6on batch data set informa(on features ML models real-(me data
- 8. Evalua8on: offline vs online
- 9. Offline evalua8on - I Training Predic6on batch real-8me Feature extrac6on batch data set features ML models real (me data informa(on Offline experiments are run on a snapshot of the collected data set.
- 10. Offline evalua8on - I PROS CONS Quick Large number of solu8ons No impact on business Applicable in most scenarios They use past data Risk to promote imita8on Not considering the impact of the algorithm on the user context Not suitable for “unpredictable” data (e.g., stock price)
- 11. Online evalua8on Training Predic6on batch real-8me Feature extrac6on batch data set features ML models real-(me data informa(on Online experiments use live user feedback
- 12. Online: human-subject experiments - I Controlled experiment A B? Human-subject experiments work in a controlled environment
- 13. Online: human-subject experiments - II PROS CONS Feedback of real users affected by actual context Implement controlled environment (back-end+front-end) Mul8ple KPIs can be measured Environment is simulated Recrui8ng non-biased users Not scaling: limited number of users Few solu8ons can be tested Mo8vate users Medium running 8me
- 14. Online: live A/B tes8ng - I A B Live tes8ng works in produc8on
- 15. Online: live A/B tes8ng - II PROS CONS Capture real, full impact of the data-driven solu8on Very few solu8ons can be tested Long running 8me Large traffic required May affect business Some KPIs are hard to measure
- 16. A/B tes8ng: under the hood Sta8s8cal hypothesis tes8ng: 1. Formulate a hypothesis 2. Set up a tes)ng campaign 3. Make use of sta)s)cs to evaluate the hypothesis
- 17. A/B tes8ng: real-world similari8es Clinical trials Product comparison Quality assurance Decision making
- 18. A/B tes8ng: UI examples ADD TO CART ADD TO CART Register Register (it’s FREE!) Lorem ipsum dolor sit amet, ius an aperiri sapientem disputando, legimus mandamus reprimique mei ea. In aliquam euripidis ius. Ei sea dico interesset. Sit et veri brute. Eu sed populo option apeirian, essent blandit ei pro. No quo integre delicatissimi. Eos ea nostro fabulas neglegentur, vel dolor splendide eu, vel ei illud blandit scripserit. Dolor detracto efficiendi ei vel. Ad per error nullam. Nec id facer impetus deseruisse. Pri dicunt phaedrum te. Ad cum munere consectetuer, has odio referrentur in. Elit atqui prodesset quo eu. Eu mei ubique bonorum deseruisse. Habeo sonet disputando et duo. Et vim homero vocibus, vel ut dicunt omnium. Start free trial Lorem ipsum dolor sit amet, ius an aperiri sapientem disputando, legimus mandamus reprimique mei ea. In aliquam euripidis ius. Ei sea dico interesset. Sit et veri brute. Eu sed populo option apeirian, essent blandit ei pro. No quo integre delicatissimi. Eos ea nostro fabulas neglegentur, vel dolor splendide eu, vel ei illud blandit scripserit. Dolor detracto efficiendi ei vel. Ad per error nullam. Nec id facer impetus deseruisse. Pri dicunt phaedrum te. Ad cum munere consectetuer, has odio referrentur in. Elit atqui prodesset quo eu. Eu mei ubique bonorum deseruisse. Habeo sonet disputando et duo. Et vim homero vocibus, vel ut dicunt omnium. Start free trial A B“Control” “Varia8on”
- 19. A/B tes8ng: ingredients Hypothesis formula8on • Everything starts with an idea Define metrics: • How to measure if something is “successful”? Run a test, collect data and compute metrics Compare the two alterna8ves
- 20. A/B tes8ng: 1) hypothesis formula8on A red bu4on is clicked more o7en than a blue bu4on Sta6s6cs lingo: Null hypothesis: There is no difference between the red and the blue buLons GOAL: reject the null hypothesis The null hypothesis is true: • we fail to reject the null hypothesis
- 21. A/B tes8ng: 2) define a metric Choose a measure that reflects your goals Examples: Click Through Rate (CTR) Open rate, click rate Conversion rate (# subs/# visitors) Customer sa8sfac8on Returning rate
- 22. A/B tes8ng: 3) run a test It may affect your business! 1. Create the two alterna)ves 2. Assign a subset of users to each alterna8ve 3. Collect data and compute the metrics
- 23. A/B tes8ng: 4) compare the two alterna8ves ADD TO CART ADD TO CART 1 view, 0 click —> 0% CTR 1 view, 1 click —> 100% CTR 100% > 0%,the red bupon is beper, right? Not so fast… A B
- 24. A/B tes8ng: confidence What is the variability of our measure? How confident are we in the outcome of the test? Model our measure resor8ng to a sta)s)cal distribu)on, e.g., a Gaussian distribu8on E.g., the average click through rate for the blue bupon is 20% ± 7% Confidence interval
- 25. A/B tes8ng: confidence interval A confidence interval is a range defined so that there is a given probability that the value of your measure falls within such range The confidence interval depends on the confidence level The higher the confidence level, the larger the confidence interval E.g., the average click through rate for the blue bupon is 20% ± 7% at 90% confidence level Confidence interval
- 26. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART
- 27. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 7% 90% confidence level
- 28. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 10% 95% confidence level
- 29. A/B tes8ng: rejec8ng the null hypothesis 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 10% 95% confidence level The avg CTR for the varia8on falls outside the CI —> Null hypothesis rejected!
- 30. A/B tes8ng: errors Null hypothesis ACCEPTED Null hypothesis REJECTED Null hypothesis TRUE True Nega)ve The buLons are the same, we acknowledge it Type I error The buLons are the same, we say the red one is beLer Null hypothesis FALSE Type II error The red buLon is beLer, we say they are the same True Posi)ve The red buLon is beLer, we acknowledge it Null hypothesis: There is no difference between the red and the blue buLons
- 31. A/B tes8ng: errors Null hypothesis ACCEPTED Null hypothesis REJECTED Null hypothesis TRUE True Nega)ve The buLons are the same, we acknowledge it Type I error The buLons are the same, we say the red one is beLer Null hypothesis FALSE Type II error The red buLon is beLer, we say they are the same True Posi)ve The red buLon is beLer, we acknowledge it Null hypothesis: There is no difference between the red and the blue buLons
- 32. A/B tes8ng: errors Null hypothesis ACCEPTED Null hypothesis REJECTED Null hypothesis TRUE True Nega)ve The buLons are the same, we acknowledge it Type I error The buLons are the same, we say the red one is beLer Null hypothesis FALSE Type II error The red buLon is beLer, we say they are the same True Posi)ve The red buLon is beLer, we acknowledge it Null hypothesis: There is no difference between the red and the blue buLons
- 33. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 7% 90% confidence level ⍺: type-I error rate
- 34. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 7% 90% confidence level β: type-II error rate
- 35. A/B tes8ng: comparing distribu8ons 20% p(CTR) CTR40% ADD TO CART ADD TO CART 20% ± 7% 90% confidence level power = 1 - β
- 36. A/B tes8ng: 8ps and common mistakes DO NOT run the two varia8ons under different condi)ons DO NOT stop the test too early Pay apen8on to external factors DO NOT blind test without a hypothesis DO NOT stop ater the first failures Choose the right metric Consider the impact on your business Randomly split the popula8on Keep the assignment consistent
- 37. Tom A/B tes8ng data-driven algorithms - I A B Training Predic6on Feature extrac6on Training Predic6on Feature extrac6on Mike People like you ChrisLena People like you Targeted Ad. Recommended users A B A B
- 38. A/B tes8ng data-driven algorithms - II CTR not always is the right metric Search engine: ideally no click at all Tweet sugges8ons: what users click is not necessarily what they want E-commerce recommenda8ons: users click to find products alterna8ve to the one proposed Find long-term metrics Reten8on/churn Returning users Time spent Upgrading users
- 39. A/B tes8ng data-driven algorithms - III Mul8ple goals are addressed Relevance Transparence Diversity Novelty Coverage Robustness Consider all the steps of the pipeline Do not vary UI and data-driven algorithm simultaneously
- 40. A/B tes8ng in the cloud - I Cloud compu8ng makes A/B tes8ng simpler: 1. Create mul8ple environments/modules with different features 2. Split traffic • e.g., Google App Engine’s traffic splivng feature Do the same with the serverless paradigm
- 41. A/B tes8ng in the cloud - II If unsure, use a third-party service A/B tes8ng as a service: • AWS A/B tes8ng service • Google Analy8cs A/B tes8ng feature • Op8mizely, VWO A/B tes8ng libraries: • Sixpack, Planout, Clutch.io, Alephbet Build your own
- 42. Data-driven algorithms to support A/B tes8ng: mul8-armed bandit - I A B A D E C D E A/B tes6ng Mul6-armed bandit CD B A F D E C B A F D E C B A F D E C B A F G Training Predic6on Feature extrac6on B A F G (me (me F
- 43. Data-driven algorithms to support A/B tes8ng: mul8-armed bandit - II PROS CONS Increased average KPI Longer 8me to reach sta8s8cal significance Harder implementa8on Harder maintain consistence
- 44. Main takeaways Evaluate data-driven solu(ons both offline and online Define the correct KPIs Prefer long-term metrics to short-term conversions Do not forget A/B tes(ng is a sta(s(cal test, rely on some cloud services if you are not “confident” Exploita(on/explora(on approaches can be an alterna(ve to A/B tes(ng Conversion rate is not the only metric
- 45. Thank you for apending :) cloudacademy.com Q & A