The document discusses deploying machine learning models in production environments. It outlines several challenges with current approaches such as models being opaque objects and a focus on training rather than prediction. It then proposes six requirements for an architecture to handle live traffic directly from trained models: 1) easy integration, 2) high performance, 3) fault tolerance, 4) scalability, 5) maintainability, and 6) extensibility. Finally, it introduces Dato Predictive Services as a platform that meets these requirements by deploying models as low-latency REST services that can elastically scale and includes monitoring and model management capabilities.

1. Deploying scikit-learn
Models in Production
Rajat Arya (@rajatarya)
Product Manager, Dato Inc.
1

2. 2
Dato provides a platform for building intelligent
apps
Data
Engineering
Data
Intelligence
Deployment
• Fast & scalable
• Rich data type support
• Visualization
• App-oriented ML
• Supporting utils
• Extensibility
• Batch & always-on
• RESTful interface
• Elastic & robust
Build, deploy, & manage your intelligent apps with Dato.

3. 3
DATA
ML
Algorithm
How Everyone Starts with ML
• Running experiments
• Plots are the results
• Not clear how to get this deployed

4. 4
DATA
ML
Algorithm
Deployment?
• Write a spec for other team to
implement in ‘production’ language
• Translate code in 6-12 months
• Stale / irrelevant model implemented
• Two teams maintaining two systems
Custom
Model
Data Engineers, Data Architects,
DevOps, App Developers
App
A
P
I
Data Scientist

5. 5
Current Challenges
• Machine Learning Models
are opaque objects
• Export format like PMML
don’t support many
models
• Focus on training, not
prediction

6. 6
Starting from the Beginning
GOAL: Handle live production traffic directly served from
the trained machine learning model
What are the requirements if we wanted to build a
similar architecture for ML Models?

7. One: Easy to Integrate
• REST APIs for both querying
and management
• Have client libraries in other
languages (no Python lock-in)
7
App
A
P
I

8. Two: High Performance
• Utilize Load Balancer for
distributing request load
• Integrated distributed cache
so repeated queries are only
answered once
8
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
Engine
A
P
I
C
A
C
H
E
LB

9. Three: Fault Tolerant
• Model running on many
machines
• System operational during
node failure
9
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
LB
GLC
Model
GLC
Model
GLC
Model
Engine
Engine
Engine

10. Four: Scalable
• Elastic scale nodes in cluster
up and down
• Easy to configure, cache
automatically updates with
cluster changes
10
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
LB
GLC
Model
GLC
Model
Engine
Engine
A
P
I
C
A
C
H
E Engine
A
P
I
C
A
C
H
E Engine

11. Five: Maintainable
• Zero downtime during model
deployment
• Metrics & logs
• Model management
11
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
LB
GLC
Model
GLC
Model
GLC
Model
Engine
Engine
Engine

12. Six: Extensible
• Arbitrary Python
• Use any set of Python
packages
• Model ensembling
12
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
LB
GLC
Model
GLC
Model
GLC
Model
Python
Python
Python

13. 13
Requirements Recap
1. Easy to Integrate
2. High Performance
3. Fault Tolerant
4. Scalable
5. Maintainable
6. Extensible
App
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
A
P
I
C
A
C
H
E
LB
GLC
Model
GLC
Model
GLC
Model
Python
Python
Python

14. 14
Do-It-Yourself
• Web Service layer:
- Tornado, Flask, Keen, Django, etc
• Caching layer:
- Redis, Cassandra, Memcached, DynamoDb, BerkeleyDb,
MySQL, etc
• Logs:
- Logback, LogStash, Splunk, Loggly
• Metrics:
- AWS CloudWatch, Mixpanel, Librato, etc

15. 15
… or use Dato Predictive Services
We set out with this goal, and used these requirements
… and now I'd like to show it to you.

16. DEMO: Deploying a scikit-learn model using
Dato Predictive Services
16

17. 17
Models as Services
• Deploy models as low-latency REST services
• Elastically scale up or out with one command
• Monitoring & Model Management
• Deploy existing Python models
• Run on AWS EC2 or Hadoop YARN
Dato Predictive Services
Predictive Engine
REST Client Direct
Model Mgmt