Keep it simple and it works - Simplicity and sticking to fundamentals in the world of big data By Mathangi Sri Head of Data Science at PhonePe
•Download as PPTX, PDF•
0 likes•110 views
With the buzz around AI and ML there is an increasing tendency for leaders and data scientists to move towards complex problem-solving. Its important to unlearn the tendency to gravitate towards complexity. In this talk we will see why avoiding complexity in ML solutions is a wiser and a quicker way to solve business problems. We can also visit some thumb rules to build pragmatic and useful models. Simplicity and sticking to fundamentals is the next "big" thing in the world of big data.
Recommended
Recommended
More Related Content
What's hot
What's hot (15)
Similar to Keep it simple and it works - Simplicity and sticking to fundamentals in the world of big data By Mathangi Sri Head of Data Science at PhonePe
Similar to Keep it simple and it works - Simplicity and sticking to fundamentals in the world of big data By Mathangi Sri Head of Data Science at PhonePe (20)
More from Analytics India Magazine
More from Analytics India Magazine (20)
Recently uploaded
Recently uploaded (20)
Keep it simple and it works - Simplicity and sticking to fundamentals in the world of big data By Mathangi Sri Head of Data Science at PhonePe
- 1. Keep it Simple And it works!
- 2. Keep it simple, stupid (KISS Principle)
- 3. OCcam’s principle of parsimony Any hypothesis that is more probable is possibly the right one to explain it
- 4. simple data science - but why?
- 5. Complexity depletes robustness Mathematical reason 1. Curse of dimensionality 2. Sparse Matrix 3. Tough to optimize 4. Overfitting Pragmatic reason 1. Danger of unstable data pipelines 2. Compute time
- 6. Simple solutions sell faster Humans need not explain how they think - but machines need to!
- 7. Use the time more to analyze the data Explore the data Create Features
- 8. You get a high return on effort Defining the right business problem Choosing the right metric Choosing right dependent variable,timeframe & data Feature creation Algorithm Hyper Parameter Tuning ML optimization funnel
- 9. Time in hand for adequate validations 1. Validations on business accuracy 2. Out of time validations 3. Thinking about any “Leakage” variables 4. Tossing to business stakeholders
- 10. simple data science - and how?
- 11. Understand status quo 1. What does the current system do today?. How is success measured
- 12. Does the problem warrant a ml solution 1. Some problems can be solved by just business rules a. For ex a hair shampoo company wants to send offers for all customers who have bought the product earlier b. Company is not averse to spend on all the people who qualify nor is it concerned by response rate 1. Sometimes even freq distribution can solve problems a. We want to predict the top 3 reasons for calling from available 10 reasons b. Top 3 reasons lets say covers 95% of the use cases c. Ordering of the reason within top 3 does not matter
- 13. Prefer supervised over unsupervised 1. Label data whenever possible 2. Instrument dependent variables if possible 3. Accuracy trade-off Vs cost of collecting samples
- 14. Prefer batch models over real time models Vs
Editor's Notes
- Kelly Johnson
- Henny-Penny the sky is falling
- Prof from George Tech
- Stakeholders find it far difficult to digest a highly complex black box models NLP & CV - people dont question Personalization or Targeting or risk based selection kind of problems people do need it to be explained Even if not the mathematical model - atleast the input features need to be explainable
- Less Labels Good number of samples per label Historic data is representative
- Most companies have systems well oiled for batch Lot of targeting/personalization/fraud/risk etc can well be solved by batch Trade off analysis - accuracy Vs complexity Keep a offset period in models