Data scientists spend a lot of time on data cleaning and munging, so that they can finally start with the fun part of their job: building models. After you have engineered the features and tested different models, you see how the prediction performance improves. However, the job is not done when you have a high performing model. The deployment of your models is a crucial step in the overall workflow and it is the point in time when your models actually become useful to your company. In this session you will learn about various possibilities and best practices to bring machine learning models into production environments. The goal is not only to make live prediction calls or have the models available as REST API, but also what needs to be considered to maintain them. This talk will focus on solutions with Python (flask, Cloud Foundry, Docker, and more) and the well established ML packages such as Spark MLlib, scikit-learn, and xgboost, but the concepts can be easily transferred to other languages and frameworks. Speaker Sumit Goyal, IBM, Software Engineer

1. IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation 1
How to deploy machine
learning models into
production
Sumit Goyal (@SumitG0yal)
Software Engineer, Watson Studio
Sven Hafeneger
Data Scientist, Watson Studio

2. IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation 2
About me
Sumit Goyal (@SumitG0yal )
Software Engineer, Watson Studio

3. Agenda
3IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

4. Motivation
4IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
Ops
Dev

5. Motivation
5IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
Work does not end with
handover of ML models !

6. Motivation
6IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation http://pobsonline.com/money-mistakes/, https://www.drwf.org.uk/news-and-events/events/100-day-challenge
Mike
Data Scientist
Deb
The Developer
ModelDev Ops
Production Environment

7. Motivation
7IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
,http://pobsonline.com/money-mistakes/, https://www.drwf.org.uk/news-and-events/events/100-day-challenge
, https://de.wikipedia.org/wiki/Zahnrad
Mike
Data Scientist
Deb
The Developer
Model + App
DevOps
Production Environment

8. Motivation
8IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
• Production environment is in the Cloud
• Python as primary programming language
• Micro-service architecture
• Python ML Frameworks
• Traditional ML algorithms, no deep learning
• Data Science Team <-> Software Development Team

9. Agenda
9IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

10. ML Model artifacts
10IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
PMML
Serialized Object
Re-Implementation
in Java/C++
Python ML Model Production Environment
App
REST
API
App

11. ML Model artifacts
11IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
PMML
Serialized Object
Re-Implementation
in Java/C++
App
REST
API
App
Python ML Model Production Environment
Handover model, code, integration
test, no re-train in Travis

12. Agenda
12IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

13. Production Environments
13IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Serialized Object
(Python)
Meta data
- Version
- Name
- Type
- ML Framework
- Eval. Metrics
- …
Python Application - REST API
Endpoint: /v1/xyz/prediction : POST
Webserver / Web
framework
Endpoint
ML framework
- Load ML model
- Create a route
- Prepare data
- Run prediction
- Post-processing
- Return result
Logic
https://www.iconsdb.com/caribbean-blue-icons/database-5-icon.html

14. Production Environments – eCommerce
14IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Endpoint
REST API
App
Service agreement
- max. response time
- availability
- quality/confidence
of prediction
- max. re-train time
- monitor
- …
https://blog.paymill.com/en/keep-e-commerce-system-functioning-optimum-capacity/,
http://www.atlascorps.org/blog/customers-and-clients-the-devils-of-business-growth/
App
App

15. Production Environments – eCommerce
15IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Endpoint
REST API
App
Service agreement
- max. response time
- availability
- quality/confidence
of prediction
- max. re-train time
- monitor
- …
https://blog.paymill.com/en/keep-e-commerce-system-functioning-optimum-capacity/,
http://www.atlascorps.org/blog/customers-and-clients-the-devils-of-business-growth/
App
App
Order == Fraud ?
Order!

16. Production Environments – Fraud Detection
16IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://www.kaggle.com/mlg-ulb/creditcardfraud
Ordering Transactions

17. Agenda
17IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

18. Deployment with Cloud Foundry
18IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Python App - REST API
Endpoint: /v1/xyz/prediction : POST
REST
- Load ML model
- Create a route
- Prep. score data
- Run prediction
- Post-processing
- Return result
Model
https://www.iconsdb.com/caribbean-blue-icons/database-5-icon.html , http://flask.pocoo.org/, http://scikit-learn.org
Questions
- How to configure the app ?
- How many instances ?
- How much RAM ?
- Zero downtime deployment ?
- Connection to DB?
- Logging ?
- Crashes and recovery?
- Routing ?
- Workload scaling ?
- Integration with other tools?
- …

19. Deployment with Cloud Foundry
19IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Choose a build pack !

20. Deployment with Cloud Foundry
20IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Python App: /route/..
http://flask.pocoo.org/, http://scikit-learn.org
../projects/fraud_detection_api
CLI: push CLI: set-env
Configuration
R
E
S
T
A
P
I
Code
App
POST: {body}
Order == Fraud ?

21. Deployment with Cloud Foundry: Demo
21IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://en.wikipedia.org/wiki/Cloud_Foundry

22. Agenda
22IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

23. Deployment with Kubernetes
23IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Container - REST API
Endpoint: /v1/xyz/prediction : POST
Endpoint
- Create a route
- Load ML model
- Prepare data
- Run prediction
- Post-processing
- Return result
Logic
https://www.iconsdb.com/caribbean-blue-icons/database-5-icon.html , http://flask.pocoo.org/, http://scikit-learn.org
Questions
- How to configure the app ?
- How many instances ?
- How much RAM ?
- How many CPUs/GPUs?
- Zero downtime deployment?
- Connection to DB?
- Logging ?
- Crashes and recovery?
- Routing ?
- Workload scaling ?
- Integration with other tools?
- …
- How much do you want to
customize the runtime?

24. Deployment with Kubernetes
24IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
Docker Container
REST
API
Logic
Docker
Images
Repository
App
(Container)
POST: {body}
https://www.iconsdb.com/caribbean-blue-icons/database-5-icon.html , http://flask.pocoo.org/, http://scikit-learn.org, https://blog.takipi.com/pivotal-
cloud-foundry-vs-kubernetes-choosing-the-right-cloud-native-application-deployment-platform/
Order == Fraud ?

25. 25IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://en.wikipedia.org/wiki/Kubernetes
Deployment with Kubernetes: Demo

26. Agenda
26IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivation
• Python ML Model artifacts
• Production Environments
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

27. Deployment with managed services
27IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
ML Models
Deployment
R
E
S
T
A
P
I
App
POST: {body}
Order == Fraud ?
Manages Deployment Service

28. Deployment with managed services
28IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://docs.microsoft.com/en-us/azure/machine-learning/, https://aws.amazon.com/sagemaker/, https://console.bluemix.net/catalog/services/machine-learning
IBM Watson Machine Learning

29. Deployment with managed services: Demo
29IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation https://console.bluemix.net/catalog/services/machine-learning

30. Summary
30IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Motivated and informed
• Deployment with Cloud Foundry
• Deployment with Kubernetes
• Deployment with managed services

31. Learn more about:
31IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation
• Cloud Foundry: https://www.ibm.com/cloud/cloud-foundry
• Kubernetes: https://kubernetes.io/
• Watson Machine Learning: https://www.ibm.com/cloud/machine-learning

32. Thank you for your attention.
Time for QA!
Sumit Goyal
sumit.goyal@de.ibm.com
32IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation