Machine Learning Model Deployment Challenges
•
14 likes•1,629 views
In this presentation I will be taking through the real world challenges of model deployment. You can check my free course below if you want to learn more on model deployment with hands on code - https://www.youtube.com/playlist?list=PL3N9eeOlCrP5PlN1jwOB3jVZE6nYTVswk
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Machine Learning Model Deployment Challenges
- 1. MODEL DEPLOYMENT Challenges and Best Practices
- 2. WHO AM I? Srivatsan Srinivasan Chief Data Scientist at Cognizant https://www.linkedin.com/in/srivatsan-srinivasan-b8131b/ https://www.youtube.com/c/AIEngineeringLife AIEngineering
- 3. Looking to Deep Dive on Model Deployment with Hands on scenario Checkout my free courses on YouTube AIEngineering Channel (Click link for details) Click Here for Link to this hands on Free Course Click Here for Link to this hands on Free Course
- 4. Complexity and Challenges with model deployment Model Deployment Options Common Mistakes to Avoid 2 1 3 4
- 5. How do most of us see Model Deployment? Model Training Model Model Pickle PMML Booster Protobuf Model Deployment Flask grpc MOJO
- 6. If model deployment was so easy.. “Launching pilots is deceptively easy but deploying them into production is notoriously challenging” – Gartner CIO Survey results "Organizations have been developing many machine learning models, but only 47% of those models makes it into production“ – Information Week • Data Integration and Strategy • Leadership Knowledge and Commitment • Model Deployment • Skill Availability • Business, Data Scientist and IT working in Silos Top 5 reasons AI fails
- 7. HOW REAL WORLD DEPLOYMENT LOOKS LIKE?
- 8. Business Case – Reduce Cart Abandonment US Based Online Retailer called “Amazing LLC” Current cart abandonment rate of 80% Industry average of cart abandonment is below 71% Tried out all options of better and faster checkout process, Support for all payment types, increase transparency etc. Hypothesize to see if cart abandonment can be reduced by 5% via targeted offers based on customer history Source : https://www.barilliance.com/cart-abandonment-rate-statistics/ Objective : Convert abandoned cart to recovered ones. Increase Revenue and Enhance Customer Experience
- 9. Simple Model Training Architecture Feature Engineering Data Preparation Model Training Model Evaluation Hyper parameter Selection/ Tuning Saved Model Data Collection Customer Transaction Campaign Data Analysis RAW + Cust_1_7_days_abandon_rate Cust_7_14_days_abandon_rate Cust_LTV Cust_Retail_abandon_rate Cust_home_abandon_rate Cust_other_abandon_rate Cust_avg_page_view_time_abandon (7,14) Cust_1_hour_avg_page_view_time Cust_3_hour_avg_page_view_time Cust_current_day_login_count Probability_to_checkout Daily/EOD Call Center/Chat Enterprise Data Store
- 10. NFR’s – Reduce Cart Abandonment 100 million customers 1 million website visits per day (2X during holidays) Peak daily volume to support 500 transaction per second (1000 tps during holidays) Time to score each customer for targeted offer display < 20 ms on web and batch scoring for targeting via email 99.95% availability. Near zero downtime
- 11. Simple Model Training Architecture Feature Engineering Data Preparation Model Training Model Evaluation Hyper parameter Selection/ Tuning Saved Model Data Collection Customer Transaction Campaign Data Analysis RAW + Cust_1_7_days_abandon_rate Cust_7_14_days_abandon_rate Cust_LTV Cust_Retail_abandon_rate Cust_home_abandon_rate Cust_other_abandon_rate Cust_avg_page_view_time_abandon (7,14) Cust_1_hour_avg_page_view_time Cust_3_hour_avg_page_view_time Cust_current_day_login_count Probability_to_checkout Daily/EOD Call Center/Chat Enterprise Data Store
- 12. Simple Model Deployment Architecture RAW+ Cust_1_7_days_abandon_rate Cust_7_14_days_abandon_rate Cust_LTV Cust_Retail_abandon_rate Cust_home_abandon_rate Cust_other_abandon_rate Cust_avg_page_view_time_abandon (7,14) Cust_1_hour_avg_page_view_time Cust_3_hour_avg_page_view_time Cust_current_day_login_count Offer_accept_flag Cart_checkedout_with_offer Average_time_spent_on_cart_post_offer Items_removed_added_post_checkout Current Session In memory/NoSQL Real Time Kafka ETL
- 13. Model Deployment in Real World Model Training Model Model Model Deployment Data Engineering Software Engineering What we thought? Reality…
- 14. Simple Model Deployment Architecture Saved ModelData Preparation Feature Engineering Serving Interface REST API Data Collection In memory/NoSQL Offline/EOD Enterprise Data Store “Data Scientist and Software Engineers in many deployment cases have different code path”
- 15. Simple Model Deployment Architecture Saved ModelData Preparation Feature Engineering Serving Interface REST API Data Collection In memory/NoSQL Functionality Testing Enterprise Data Store “Data Scientist and Software Engineers in many deployment cases have different code path” Functionality Testing
- 16. Model Deployment in Real World Model Training Model Model Model Deployment Data Engineering Software Engineering What we thought? Reality… Functionality Testing
- 17. Simple Model Deployment Architecture Saved ModelData Preparation Feature Engineering Serving Interface REST API Data Collection In memory/NoSQL Offline/EOD 20 ms Enterprise Data Store 1 User 10 User 100 User 1000 User Performance Testing NFR < 20 ms response time 1000 tps volume Highly Available
- 18. Typical Resource Bottleneck Shared Resources – (CPU, Memory, Bandwidth etc) Garbage Collection Network bottleneck Memory Management Shared Resources – (Network switches and Shared file system) CPU starvation Daemons and Background Activity
- 19. Model Deployment in Real World Model Training Model Model Model DeploymentData Engineering Software Engineering What we thought? Reality… Functionality Testing Load/Performance Testing
- 20. Model Deployment - Scaling Flask Application Saved ModelData Preparation Feature Engineering Serving Interface REST API Data Collection In memory/NoSQL
- 21. Model Deployment - Scaling VM/Containerize Flask ApplicationGunicornNgnix
- 22. Model Deployment – Scaling (Virtual Machine) Flask ApplicationGunicorn Ngnix Flask ApplicationGunicorn Flask ApplicationGunicorn Flask ApplicationGunicorn Ngnix Load balancer
- 23. Model Deployment - Scaling Scale up to 1000 tps Scale down to 500 tps
- 24. Model Deployment in Real World Model Training Model Model Model DeploymentData Engineering Software Engineering What we thought? Reality… Functionality Testing Load/Performance Testing Infrastructure
- 25. Infrastructure Bare Metal Infra vs Kubernetes vs Cloud CPU vs GPU Number of Servers required to Scale to peak volume Cores and Memory per server Disaster Recovery/HA – Active/Active or Active/Passive
- 26. Remember: “Infrastructure cost can play key Factor in application with low latency and high availability need”
- 27. Model Deployment in Real World Model Training Model Model Model DeploymentData Engineering Software Engineering What we thought? Reality… Functionality Testing Load/Performance Testing Infrastructure CI/CD Pipeline
- 28. Be Prepared … Model Monitoring • Model performance can deteriorate any time no matter how good your model training performance is • In some cases it can be as soon as you start testing with real world data
- 29. Model Monitoring
- 30. Simple Model Deployment Architecture Saved ModelData Preparation Feature Engineering Serving Interface REST API Data Collection In memory/NoSQL Offline/EOD Enterprise Data Store Model Monitoring Real Time Dashboard (Business and Model KPI)
- 31. Model Deployment in Real World Model Training Model Model Model DeploymentData Engineering Software Engineering What we thought? Reality… Functionality Testing Load/Performance Testing Infrastructure CI/CD Pipeline Model Monitoring
- 32. Model Deployment – Zero downtime deployment Model Version 1 Image Source: https://www.ianlewis.org/en/bluegreen-deployments-kubernetes
- 33. Model Deployment – Zero downtime deployment Model Version 1 Model Version 2
- 34. Model Deployment – Zero downtime deployment Model Version 1 Model Version 2
- 35. Model Deployment – Champion Challenger Deployment Champion Challenger90% 10%
- 36. Model Deployment in Real World Model Training Model Model Model DeploymentData Engineering Software Engineering What we thought? Reality… Functionality Testing Load/Performance Testing Infrastructure CI/CD Pipeline Model Monitoring MLOps
- 37. Simple Model Deployment Architecture Application/Infrastructure Monitoring MLOps (Model Monitoring, Management)
- 40. Deployment Patterns Batch Near Real Time Real Time Edge Simple Hard
- 41. BEST PRACTICES
- 42. Best Practice • Start model deployment and integration understanding along with business problem framing • Have right participation from business, machine learning, data engineering and software engineering team during framing session • Keep model pipelines as simple as possible and as long as possible • Balance between performance and simplicity where possible • Plan for Infrastructure at an very early stage of the project • Invest in building cross project capability • Model Monitoring • Setting up CI/CD pipeline • Model Retraining automation • Feature Store
- 43. THANK YOU