Building and deploying a machine learning model can be difficult to do once. Enabling other data scientists (or yourself, one month later) to reproduce your pipeline, to compare the results of different versions, to track what’s running where, and to redeploy and rollback updated models is much harder. In this talk, I’ll introduce MLflow, a new open source project from Databricks that simplifies the machine learning lifecycle. MLflow provides APIs for tracking experiment runs between multiple users within a reproducible environment, and for managing the deployment of models to production. MLflow is designed to be an open, modular platform, in the sense that you can use it with any existing ML library and development process. MLflow was launched in June 2018 and has already seen significant community contributions, with over 50 contributors and new features including language APIs, integrations with popular ML libraries, and storage backends. I’ll show how MLflow works and explain how to get started with MLflow.

1. : Accelerating the
End-to-End ML Lifecycle
Matei Zaharia
@matei_zaharia

2. 2
About Databricks
Founded in 2013 by the original creators of Apache Spark
Unified analytics platform for data
science, engineering and AI
• Optimized Spark and ML runtime
• Collaborative & secure workspace
Apache Spark, Spark and Apache are trademarks of the Apache Software Foundation

3. Some of Our Customers
Financial Services Healthcare & Pharma Media & Entertainment Technology
Public Sector Retail & CPG Consumer Services Energy & Industrial IoTMarketing & AdTech
Data & Analytics Services

4. Some of Our Customers
Financial Services Healthcare & Pharma Media & Entertainment Technology
Public Sector Retail & CPG Consumer Services Energy & Industrial IoTMarketing & AdTech
Data & Analytics Services
Correlate EMR of 50,000 patients
compared with their DNA

5. Some of Our CustomersFinancial Services Healthcare & Pharma Media & Entertainment Technology
Public Sector Retail & CPG Consumer Services Energy & Industrial IoTMarketing & AdTech
Data & Analytics Services
Provide recommendations to sales
using NLP and deep learning

6. Some of Our Customers
Financial Services Healthcare & Pharma Media & Entertainment Technology
Public Sector Retail & CPG Consumer Services Energy & Industrial IoTMarketing & AdTech
Data & Analytics Services
Curb abusive behavior
across gamers globally

7. Machine Learning
Development is Complex

8. ML Lifecycle
8
Delta
Data Prep
Training
Deploy
Raw Data
μ
λ θ Tuning
Scale
μ
λ θ Tuning
Scale
Scale
Scale
Model
Exchange
Governance

9. “I build 100s of models/day to lift revenue, using any library:
MLlib, PyTorch, R, etc. There’s no easy way to see what data
went in a model from a week ago and rebuild it.”
-- Chief scientist at ad tech firm
Example

10. Example
“Our company has 100 teams using ML worldwide. We can’t
share work across them: when a new team tries to run some
code, it doesn’t even give the same result.”
-- Large consumer electronics firm

11. Custom ML Platforms
Facebook FBLearner, Uber Michelangelo, Google TFX
+Standardize the data prep / training / deploy cycle:
if you work within the platform, you get these!
– Limited to a few algorithms or frameworks
– Tied to one company’s infrastructure
Can we provide similar benefits in an open manner?

12. www.mlflow.org
Introducing
Open source machine learning platform
• Works with any ML library & language
• Runs the same way everywhere & cross-cloud
• Scales to big data with Apache Spark
Launched this June
• Already >50 contributors and many new features!

13. MLflow Design Philosophy
1. API-first, “open-interface” platform
• Allow submitting runs, models, etc from any library & language
– Example: a “model” can just be a Python lambda function
• Key enabler: built around REST APIs and CLI
2. Modular design
• Let users mix & match components in their existing workflows
• Key enabler: multiple components that can be used separately

14. MLflow Components
14
Tracking
Record and query
experiments: code,
data, config, results
Projects
Package workflows
into reproducible
and reusable steps
Models
Model packaging
format for diverse
deployment tools

15. Model Development without MLflow
data = load_text(file)
ngrams = extract_ngrams(data, N=n)
model = train_model(ngrams,
learning_rate=lr)
score = compute_accuracy(model)
print(“For n=%d, lr=%f: accuracy=%f”
% (n, lr, score))
pickle.dump(model, open(“model.pkl”))
For n=2, lr=0.1: accuracy=0.71
For n=2, lr=0.2: accuracy=0.79
For n=2, lr=0.5: accuracy=0.83
For n=2, lr=0.9: accuracy=0.79
For n=3, lr=0.1: accuracy=0.83
For n=3, lr=0.2: accuracy=0.82
For n=4, lr=0.5: accuracy=0.75
...
What if I expand
the input data?
What if I tune this
other parameter?
What if I upgrade
my ML library?
What version of
my code was this
result from?

16. Model Deployment without MLflow
Code & Models
DATA
SCIENTIST
PRODUCTION
ENGINEER
Please deploy this
SciKit model!
Please deploy this
Spark model!
Please deploy this
R model!
Please deploy this
TensorFlow model!
Please deploy this
ArXiv paper!
…

17. Model Development with MLflow
data = load_text(file)
ngrams = extract_ngrams(data, N=n)
model = train_model(ngrams,
learning_rate=lr)
score = compute_accuracy(model)
print(“For n=%d, lr=%f: accuracy=%f”
% (n, lr, score))
pickle.dump(model, open(“model.pkl”))

18. $ mlflow ui
Model Development with MLflow
data = load_text(file)
ngrams = extract_ngrams(data, N=n)
model = train_model(ngrams,
learning_rate=lr)
score = compute_accuracy(model)
mlflow.log_param(“data_file”, file)
mlflow.log_param(“n”, n)
mlflow.log_param(“learning_rate”, lr)
mlflow.log_metric(“score”, score)
mlflow.sklearn.log_model(model)
Track parameters, metrics,
output files & code version
Search using UI or API

19. MLflow UI: Inspecting Runs

20. MLflow UI: Comparing Runs

21. Project Spec
Code ConfigDeps
Local Execution
Remote Cluster
Packaging Code: MLflow Projects
$ mlflow run git://...

22. Example MLflow Project
my_project/
├── MLproject
│
│
│
│
│
├── conda.yaml
├── main.py
└── model.py
...
conda_env: conda.yaml
entry_points:
main:
parameters:
training_data: path
lambda: {type: float, default: 0.1}
command: python main.py {training_data} {lambda}
$ mlflow run git://<my_project>
mlflow.run(“git://<my_project>”, ...)

23. Model Format
ONNX Flavor
Python Flavor
Training Apps
Batch & Stream Scoring
REST Serving Tools
Packaging Models: MLflow Models
Packaging Format
. . .
Inference Code
MLlib

24. Example MLflow Model
my_model/
├── MLmodel
│
│
│
│
│
└── estimator/
├── saved_model.pb
└── variables/
...
Usable by tools that understand
TensorFlow model format
Usable by any tool that can run
Python (Docker, Spark, etc!)
run_id: 769915006efd4c4bbd662461
time_created: 2018-06-28T12:34
flavors:
tensorflow:
saved_model_dir: estimator
signature_def_key: predict
python_function:
loader_module: mlflow.tensorflow
$ mlflow pyfunc serve -r <run_id>
spark_udf = pyfunc.spark_udf(<run_id>)

25. Model Deployment with MLflow
DATA
SCIENTIST
PRODUCTION
ENGINEER
Please deploy this
MLflow Model!
OK, it’s up in our REST
server & Spark!
Please run this
MLflow Project
nightly for updates!
Don’t even tell me
what ArXiv paper
that’s from...

26. MLflow Development Status
Many new features since our release in June
• Model packaging for MLlib, H2O, TensorFlow, PyTorch, Keras
• R API (contributed by RStudio)
• Java & Scala API
• Storage backends: Azure, AWS, Google, (S)FTP
Just released MLflow 0.8.0
• Multi-step workflow UI
• Compact table view
• Azure ML Serving Workspace deploys

27. Early MLflow Use Cases
Uses MLflow Tracking to build & monitor
hundreds of models for entities in energy grid
Uses MLflow Projects to package and run
reproducible deep learning jobs in the cloud
Uses MLflow Models to package and deploy a
recommendation model + custom logic
European
Energy Company
Marketplace
Online Retailer

28. Conclusion
Machine learning platforms can simplify ML development
for both data scientists and engineers
To get started with MLflow, just pip install mlflow
Docs & tutorials at mlflow.org
+See Jules Damji’s talk tomorrow at 11!