Active Learning and Human-in-the-Loop
•Download as PPTX, PDF•
4 likes•1,982 views
Human-in-the-loop machine learning is an incredibly valuable design pattern used in real world machine learning deployments across many types of applications. Taking the best of what humans can do and combining that with the best computers is practical and powerful. Not only that, but it mimics a strategy called active learning, forcing the training data collection process to become very efficient so the algorithm gets better and better.
Recommended
Recommended
More Related Content
Similar to Active Learning and Human-in-the-Loop
Similar to Active Learning and Human-in-the-Loop (20)
More from CrowdFlower
More from CrowdFlower (11)
Recently uploaded
Recently uploaded (20)
Active Learning and Human-in-the-Loop
- 2. The Effect of Better Algorithms 0% 5% 10% 15% 20% 25% Naïve Bayes Maximum Entropy SVM Classifier Error Rate Active Semi-Supervised Learning for Improving Word Alignment (Vamshi ACL ’10) Real World Data 2
- 3. The Effect of Better Features 0% 5% 10% 15% 20% 25% 30% Unigrams Bigrams Unigrams+Bigrams Classifier Error Rate 3
- 4. The Effect of More Data Active Semi-Supervised Learning for Improving Word Alignment (Vamshi ACL ’10) Real World Data 0% 2% 4% 6% 8% 10% 12% 14% N 2N 4N Classifier Error Rate 4
- 5. The Effect of Cleaner Data 0% 2% 4% 6% 8% 10% 12% 14% 90% Accurate Data 95% Accurate Data 100% Accurate Data Classifier Error Rate 5
- 6. Where Do Data Scientists Spend Their Time? 6 Source: CrowdFlower Data Science Report 2015
- 7. CrowdFlower Data Enrichment Platform 7
- 8. Color Data 8
- 9. 9
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. 14
- 15. Apple Watch 15
- 16. Apple Watch 16
- 17. Apple Watch 17
- 18. Apple Watch 18
- 19. Collecting the Same Data Over and Over 19
- 20. Open Data 20
- 21. Make Your Data Public Setting 21
- 23. Data For Everyone Library 23
- 26. Open Data API 26
- 29. Record Data 29
- 30. Extracting Names and Titles 30
- 31. Summarization 31
- 32. Is an Image Funny? 32
- 35. 35
- 36. Kaggle accuracy 0% 10% 20% 30% 40% 50% 60% 70% Baseline 12-May 13-May 14-May 15-May Accuracy Accuracy of Best Performing Model 36
- 37. Kaggle accuracy over time 0% 10% 20% 30% 40% 50% 60% 70% 80% Accuracy Accuracy of the Best Performing Model 37
- 38. Kaggle Participation 0 200 400 600 800 1000 1200 1400 Number of Participating Teams 38
- 39. AI Classifier Output Confident Human in the Loop 39
- 40. Human in the Loop Confident Output AI Classifier Human Annotation 40
- 41. Human in the Loop Confident Output AI Classifier Active Learning Human Annotation 41
- 42. Human in the Loop Confident Output AI Classifier Active Learning Human Annotation 42
- 43. Active Learning From hunch.net active learning tutorial ICML ‘09 43
- 44. Active Learning Accuracy Improvement 44
- 45. Google Cars Miles Per Disengage 45
- 46. Adaptive Cruise Control Image source: ExtremeTech 46
- 47. Advanced Chess Image source: Computer Chess 47
- 48. AlphaGo 48
Editor's Notes
- Over 200,000 Records
- 59,000 records
- Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence - Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t - Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough
- I think we can and should apply this to every business process We start with a machine learning classifier. Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence - Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t - Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time. We didn’t invent any of this, lot’s of people are talking about this and thinking about this, including many people in the room. But looking at the industry we see a lot more people talking about it than actually doing it. We are going to make this setup so easy that you will have no excuse for not doing it.
- I think we can and should apply this to every business process We start with a machine learning classifier. Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence - Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t - Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time. We didn’t invent any of this, lot’s of people are talking about this and thinking about this, including many people in the room. But looking at the industry we see a lot more people talking about it than actually doing it. We are going to make this setup so easy that you will have no excuse for not doing it.
- A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time.
- handed control to the driver 272 times and a test driver felt compelled to intervene 69 times
- In the field of chess computers passed humans a long time ago. But if you really want to make a great chess playing algorithm you would still use a human and computer together. There is a subculture of folks who still play “Advanced Chess” and this is actually where the highest quality chess games take place. - Still situations where humans are better