SlideShare a Scribd company logo
1 of 50
Download to read offline
2
Machine Learning
Platformization & AutoML
Adopting ML at scale in the Enterprise
Ed Fernandez
@efernandez
3
Disclaimer:
The term AI (Artificial Intelligence) appears several times throughout these slides in several references and 3rd party content
In the context of this presentation it refers specifically to the ability to build machine learning driven applications which
ultimately automate and/or optimize business processes and DOES NOT refer to true or strong Artificial Intelligence in the
formal sense, which is not likely to happen for decades to come (emphasis from author)
1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf
2. ML Systems - Jeff Smith (book)
3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773
4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/
5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/
7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/citation.cfm?id=2843948.
8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on-airbnb-9272d3d4739d.
9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management-platform.
10. Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone.
11. Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018-facebook.pdf.
12. Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/.
13. Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007.
14. BigML Documentation https://bigml.com/documentation/
15. Domain Specific Language for ML Workflows Automation - WhizzML - BigML https://bigml.com/whatsnew/whizzml#whizzml-automating-machine-learning
16. Domain Specific Language for Feature Engineering - Flatline https://github.com/bigmlcom/flatline
17. AutoML - OptiML https://bigml.com/api/optimls
References (Partial List):
#MLSEV
[1] Definitions & Context
•Machine Learning Platforms, Definitions

•ML models & apps as first class assets in the Enterprise

•Workflow of an ML application

•ML Algorithms, overview

•Architecture of a ML platform

•Update on the Hype cycle for ML & predictive apps

[2] Adopting ML at Scale
•The Problem with Machine Learning - Scaling ML in the
Enterprise

•Technical Debt in ML systems

•How many models are too many models

•The need for ML platforms 

[3] The Market for ML Platforms
•ML platform Market References - from early adopters to
mainstream

•Custom Build vs Buy: ROI & Technical Debt

•ML Platforms - Vendor Landscape

4
Summary
[7] Future Evolution for ML Platforms
Appendix I: Practical Recommendations for ML onboarding in the
Enterprise

Appendix II: List of References & Additional Resources

[4] Custom Built ML Platforms
• ML platform Market References - a closer look

Facebook - FBlearner

Uber - Michelangelo

AirBnB - BigHead

• ML Platformization Going Mainstream: The Great Enterprise Pivot

[5] From DevOps to MLOps
• DevOps <> ModelOps

• The ML platform driven Organization

• Leadership & Accountability (labour division)

[6] Automated ML - AutoML
• Scaling ML - Rapid Prototyping & AutoML:

• Definition, Rationale

• Vendor Comparison

• AutoML - OptiML: Use Cases
#MLSEV
•
Machine Learning Platforms, Definitions
•
ML use cases & apps as first class assets in the Enterprise
•
Workflow of an ML application
•
ML Algorithms, overview
•
Architecture of an ML platform
•
Update on the Hype cycle for ML & predictive apps
5
Definitions & Context
Section 1
6
The ML platform offers advanced functionality essential for building ML solutions (primarily predictive and prescriptive models). 

The platform supports the incorporation of these solutions into business processes, surrounding infrastructure, products and applications. 

It supports variously skilled data scientists (and other stakeholders i.e ML engineers, Data Analysts & Business Analysts and experts) in
multiple tasks across the data and analytics pipeline, including all of the following areas: 

• Data ingestion 

• Data preparation & Transformation

• Data exploration & Visualization

• Feature engineering



• Model Selection, Evaluation & testing (AutoML)

• Deployment 

• Monitoring 

• Maintenance 

• Collaboration 

Machine Learning Platforms
A formal definition
The workflow of a machine learning project. Defining a problem, prototyping a solution, productionizing the solution and measuring the impact of
the solution is the core workflow. The loops throughout the workflow represent the many iterations of feedback gathering needed to perfect the
solution and complete the project.
Adapted from Gartner DSML Data Science and Machine Learning Platforms report, February 2020 - ID G00385005
Internal &
External
AI assets:
ML modeling,
heuristics
AI assets:
ML platform
AI assets:
People
skills/expertise
ML Adoption
cross-function
Enterprise Roadmap for AI & ML
ML models as first-class enterprise asset
SUPERVISED UNSUPERVISED
DATA Requires “labelled” data Does not require “labelled” data
GOAL
Goal is to predict the label often called the objective
(churn, sales predictions, etc).
Goal is “structure discovery”, with
algorithms focused on type of relation
(clustering, etc.)
EVALUATION Predictions can be compared to real labels
Each algorithm has it’s own quality
measures
ALGORITHMS
ML Algorithms
8
CLUSTER ANOMALY
TOPIC
MODEL
ASSOCIATIONTREE
MODEL
ENSEMBLE DEEPNETLOGISTIC
REGRESSION
TIME SERIES
CLASSIFICATION / REGRESSION
OPTIML
9
Deep Learning:
Specific Use
Cases
ANN
CNN & RNN
Bayesian NN
(traditional)
Machine Learning:
Workhorse
algorithms
Linear & Logistic
Regression
Decision Trees &
Random Forest
Ensembles
source: Kaggle · The State of Data Science
& ML 2019 ·
https://www.kaggle.com/kaggle-survey-2019
Machine Learning Adoption
ML Algorithms in practice
BigML, Inc
Where are my models?
10
Architecture of a ML Platform - MLaaS - BigML
• Models are stored in the BigML server, in the cloud.
• Private and On premises clouds are also available.
• API first: every execution (model, dataset,
evaluation, automation script) is an immutable
resource that can be managed programmatically.
• Resources are encoded in JSON. are easy to
integrate and export to other applications and
workflows
API-first, auto-scalable, auto-deployable
distributed architecture for Machine Learning
A.I.
you & I
Emerging Technology hype cycle: Machine Learning
The Great Enterprise Pivot
We are here
~2 years to
mainstream
Adoption Cycle: Machine Learning
Custom Built vs Buy, crossing the chasm
source: adapted from BigML Inc materials · http://bigml.com
We are here
• Open
Source
• Custom Built
vs Buy
• Fragmented
• Proprietary
• Buy vs Build
• Consolidated
#MLSEV
The Problem with Machine Learning - Adopting ML at Scale in the
Enterprise
Technical Debt in ML systems
How many models are too many models
The need for ML platforms
14
Adopting ML at Scale
Section 2
“
The problem with Machine Learning
Adopting Machine Learning at Scale in the Enterprise
It is time to bring the AI exploration
era to the next stage of production -
enabling sustainable, industrial-
grade AI systems within the IT and
cultural fabric.
Gartner
“Artificial Intelligence Primer for 2020” Erick Brethenoux, 24 January 2020
16
The problem with Machine Learning
source: Kaggle · The State of Data Science &
ML 2019 ·
https://www.kaggle.com/kaggle-survey-2019
From prototyping to production
17
D. Sculley et al., Google, NIPS 2015
Technical Debt in Machine Learning
Model Drifting - Data Lifecycle
18
NIPS: Hidden technical debt in ML
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
Dealing with Complexity
Infrastructure & Fragmentation
19
How many ML models are too many models
Facebook ML platform (a.k.a FBlearner):
+1Mn ML models trained
+6 Mn predictions/sec
25% of engineering team using it
Source: ModelOps IBM research Waldemar Hummer et al http://hummer.io/docs/2019-ic2e-modelops.pdf
20
Source: David Talby CTO, Pacific AI - Strata Conference
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68616
Increasing number of models & complexity
ML Use Cases
21
Increasing number of models & complexity
Uber
Facebook
Twitter
Linkedin
SO PUT THE RIGHT ML PLATFORM IN PLACE
THESE COMPANIES DID ALREADY (Custom Built)
•e-commerce
•online/real time
transaccions
•consumer C2C services
•Predictions driven by
volume (millions) & models
•long term trends &
patterns
•B2B & Government
services
•consumer C2C services
•Predictions driven by
quality &
•rules based knowledge
AirBnB
Lyft
Netflix
Spotify
GE
AT&T
eBay
Amazon
#MLSEV
ML platform Market References - from early adopters to mainstream
Custom Build vs Buy: ROI & Technical Debt
ML Platforms - Vendor Landscape
22
The Market for ML Platforms
Section 3
Amazon
Jeff Bezos’ letter to Amazon shareholders - May, 2017
“Machine learning and AI is a horizontal
enabling layer. It will empower and improve
every business, every government
organization, every philanthropy — basically
there’s no institution in the world that cannot
be improved with machine learning” .
Jeff Bezos
Machine Learning Platforms
An Infrastructure & Service layer to drive ML at scale in the enterprise
Facebook FBlearner May 9, 2016
https://code.fb.com/core-data/
introducing-fblearner-flow-facebook-s-
ai-backbone/
Google TFX Tensorflow Aug 13, 2017
https://www.tensorflow.org/tfx/
https://dl.acm.org/ft_gateway.cfm?
id=3098021&ftid=1899117&dwn=1&CF
ID=81485403&CFTOKEN=79729647b
2ac491f-EAC34BCC-93F2-A3C5-
BE9311C722468452
Netflix
Notebook Data
Platform
Aug 16, 2018 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Uber Michelangelo Sept 5, 2017 https://eng.uber.com/michelangelo/
Twitter Cortex Sept, 2015
https://cortex.twitter.com/en.html
https://blog.twitter.com/engineering/
en_us/topics/insights/2018/ml-
workflows.html
Magic Pony acquisition - 2016:
https://www.bernardmarr.com/
default.asp?contentID=1373
AirBnB BigHead Feb, 2018
https://databricks.com/session/
bighead-airbnbs-end-to-end-machine-
learning-platform
LinkedIN Pro-ML Oct, 2018
https://engineering.linkedin.com/blog/
2018/10/an-introduction-to-ai-at-
linkedin
an unfair ‘platform’ advantage
Machine Learning Platforms
eBay Krylov Dec 17, 2019
https://tech.ebayinc.com/engineering/
ebays-transformation-to-a-modern-ai-
platform/
Lyft Flyte Jan 20, 2020
https://eng.lyft.com/introducing-flyte-
cloud-native-machine-learning-and-
data-processing-platform-
fb2bb3046a59
AT&T Acumos Oct 30, 2017 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Spotify
Spotify ML
platform
Dec 13, 2019
https://labs.spotify.com/2019/12/13/the-
winding-road-to-better-machine-
learning-infrastructure-through-
tensorflow-extended-and-kubeflow/
Delta Airlines (licensed) Jan 8, 2020
https://www.aviationtoday.com/
2020/01/08/delta-develops-ai-tool-
address-weather-disruption-improve-
flight-operations/
GE
Predix (customer
IoT platform)
Feb, 2018
https://www.ge.com/digital/sites/
default/files/download_assets/Predix-
The-Industrial-Internet-Platform-
Brief.pdf
KT Telecom Neuroflow Jan, 2018 https://disruptive.asia/kt-ai-platform-
internal-use/
An Infrastructure & Service layer to drive ML at scale in the enterprise
Machine Learning Platforms
Build vs Buy
The “custom build” approach, while highly customized to the needs of the organization, is
expensive, requires time and strong engineering talent and teams to develop and maintain it

The “buy” option often requires adapting to a given vendor’s approach but demands less time and
expertise and provides continued access to innovations
Ultimately, it’s a business case decision (ROI calculator next slide)
Partial list of ML platform licensees (courtesy of BigML Inc)
Most enterprises will ultimately implement
their ML platforms from commercial or
cloud-delivered software, 

along with custom integration and custom-
coded modules tailored to their specific
needs
28
ML Platform 

Build vs Buy ROI

Source: Dataiku DS ROI toolkit https://pages.dataiku.com/data-science-roi-toolkit
29
MACHINE LEARNING AS A SERVICE MACHINE LEARNING PLATFORM & SOFTWARE
https://www.crisp-research.com/vendor-universe/machine-learning/#fndtn-mlaas
Machine Learning Platforms
Vendor Landscape MLaaS: Machine Learning as a Service & On Premise
30
ML Platformization Going Mainstream
Buy vs Build
Partial list of ML platform public customer references: HG Insights (BigML Inc, Dataiku & H2O.ai https://discovery.hgdata.com/product/bigml)
#MLSEV
ML platform Market References - a closer look
•Facebook - FBlearner
•Uber - Michelangelo
•AirBnB - BigHead
ML Platformization Going Mainstream: The Great Enterprise Pivot
31
Custom Built ML Platforms
Section 4
Facebook
FBlearner Flow: Facebook’s ML platform for internal use - May, 2016
25% of engineering team
using it
+1Mn ML models trained
+6 Mn predictions/sec
ML at scale:
Reusability
Parallelization
Simplicity
Automation
Rapid prototyping & experimentation
Facebook
FBlearner Flow: Facebook’s ML platform for internal use - May, 2016
Eliminating manual work for
experimentation
Engineers can spend more time
on feature engineering
which in turn produce greater
accuracy improvements
“
Uber
Michelangelo: Uber’s MLaaS platform for internal use - Sept, 2017
end-to-end ML workflow:
• manage data
• train
• evaluate
• deploy models
• make and monitor predictions.
Supports traditional ML models,
time series forecasting, and deep
learning.
35
Uber
Michelangelo: Uber’s MLaaS platform for internal use - Sept, 2017
AirBnB
Bighead - Feb, 2018
Airbnb’s internal ML platform is called Bighead.
Bighead is an end-to-end platform for building and deploying ML
models that aims to make the machine learning process at Airbnb
seamless, versatile, consistent, and scalable.
It is built in Python and relies on open source technology like
Docker, Jupyter, Spark, Kubernetes, and more.
These open source components are customized and integrated for
Airbnb’s specific needs. Like much of Airbnb’s technology
infrastructure, Bighead runs in AWS.
The platform was supported by an ML infrastructure team of 11
engineers and one product manager.
In the fall of 2018, Airbnb announced its plans to open source parts
of Bighead and Zipline in early 2019, but this hasn’t yet materialized.
The Great Pivot - ML at scale
Systems of Intelligence/ML drive efficiencies (1st), competitive advantages (2nd) & next
defensible business models ultimately
• Most large technology companies are
reconfiguring themselves around ML.
• Google was (arguably) the first company to
move, followed by Microsoft, Facebook,
Amazon, Apple and IBM.
• 2nd tier corporations following suit: GE, Uber,
even carriers as AT&T
• Not only a US phenomena - Alibaba, Baidu
chief Robin Li said in an internal memo that
Baidu’s strategic future relies on AI
• Ultimately all global players will need to re-tool
their processes adopting a ML driven
approach.h/t Jerry Chen - Greylock Partners
https://news.greylock.com/the-new-moats-53f61aeac2d9
#MLSEV
Scaling ML - Rapid Prototyping & AutoML:
Definition, Rationale
Vendor Comparison
AutoML - OptiML: Use Cases
38
Automated ML - AutoML
Section 6
“All Models are wrong, but some are useful”
AutoML
Automated Machine Learning
40
Problem Formulation
Data Acquisition
Feature Engineering
Modeling and Evaluations
Predictions
Measure Results
Data Transformations
5%
80%
• Data tasks, most consuming - Semi
automated.
• Feature Engineering is key to model
performance - semi automated
10% • Goal definition - Human driven
5%
• AutoML enables fast modeling/prototyping -
Automated
• Automated
41
Enable knowledge workers (e.g., analysts, developers) to build stable and
insightful models quickly 

Scale the number of predictive use cases in collaboration with non-technical
peers through quick prototyping.
Best AutoML approaches rely on automation of parts of the Machine Learning
process (e.g., hyper-parameter tuning) without limiting the practitioners’ ability
control customization. 

GDPR, data privacy, interpretability and prediction explanations became
critical concerns when deploying AutoML
AutoML
Automated Machine Learning
AutoML
42
Typical AutoML pipeline
AutoML
Feature
generation
Feature
selection
Model
selection= + +
43
AutoML
Trade off in Model/Algorithm Selection
• Simple (Logistic Reg) vs
Complex (Deepnets, ANNs) 

• Weak and Fast vs. Slow and
Robust

• Interpretability vs.
Representability 

• Confidence vs. Performance 

• Biased vs. Data-hungry
44
AutoML DATAROBOT H2O BigML
Data Preparation
• Encoded categorical variables (one-hot);
Text n- grams; Missing values imputing;
Discretization (bins) 

• limited manual transformations • Max. of
10 classes in the objective*
•Encoded categorical variables (one-hot); Missing
values handling; Date-time fields expansion; Bulk
interactions transformers; SVD numeric
transformer; CV target encoding; Cluster distance
transformer; Time lag 

•Automatic feature engineering possible when
using AutoDL
• Encoded categorical variables (one-hot); Text
analysis; Missing values handling; Date-time fields
expansion 

• Automatic Recursive Feature Selection & Feature
Engineering

• Multiple flexible manual transformations • Max of
1,000 classes in the objective
Optimization
Undisclosed optimization technique 

(“expert data scientists preset
hyperparameter search space for models*)
Random Stacking 

(a combination of random grid search and stacked
ensembles, plus early stopping)
Bayesian Parameter Optimization 

(SMAC — Sequential Model-based Algorithm
Configuration) & DNN Metalearning
Models
•Open-source libraries: scikit-learn, R, H2O,
Tensorflow (not CNN or RNN), Spark,
XGBoost, DMTK, and Vowpal Wabbit 

•They also “blend” multiple models during
the optimization process.
•GBMs, Random Forests, XGBoost, deep neural
nets, and extreme random forests 

•· Stacks of models can be learned. Best of family
stacks adopt the top model type from each of the
main algorithms.
•Decision trees, random decision forests, boosting,
logistic regression, deep neural networks 

•Customizable model ensembles with Fusions
leveraging the individually optimized models for
different classification, regression algorithms.
Speed It tests 30-40 different modeling
approaches and takes ~20 min.
Default time limit for AutoML is 1 hour. Can use
GPU or CPU. Can specify settings for accuracy,
time, and interpretability.
It tests 128 different modeling approaches
(creating more than 500 resources) and takes ~30
min.
Model
Visualizations &
Interpretability
• Limited model visualizations 

• Feature importance for models • Predictions
explainability
• Dashboard: A single page with a global
interpretable model explanations plot, a feature
importance plot, a decision tree plot, and a partial
dependence plot. 

• A machine learning interpretation tool (MLI) that
includes a KLIME or LIME-SUP graph.
• Multiple model visualizations to analyze the
impact of the variables on predictions:
sunburst, decision tree, partial dependence
plots, line chart (LR)
• Feature importance for models
• Predictions explainability
Model Evaluations
• Confusion matrix

• ROC curve (only for binary classification)

• Lift curve (only for binary classification)

• Side-by-side evaluations comparison

• Trade-off between complexity vs.
performance
• Models are ranked by cross-validation 

AUC by default. 

• Return leaderboard sortable by deviance (mean
residual deviance), logloss, MSE, RMSE, MAE,
RMSLE, mean per class error
• Confusion matrix

• ROC curve

• Precision-Recall curve

• Gain curve

• Lift curve

• Multiple evaluations comparison chart
Programmability &
Deployability
• Models can be used and created via API •
Export models

• Cloud, VPC or on-premises
• H2O allows you to convert the models you have
built to either a Plain Old Java Object (POJO) or a
Model ObJect, Optimized (MOJO). 

• H2O-generated MOJO and POJO models are
ieasily embeddable in Java environments
• Models can be used and created via API • Export
models

• Cloud, VPC or on-premises
Source: Public Resources, Vendor Docs, BigML Analysis
Metalearning!
45
AutoML - Metalearning
Automatic Network Hyperparameters Selection - DNNs (DeepNets)
We trained 296,748 deep neural networks
so you don’t have to!
• 296,748+ deep neural networks trained on 50 datasets
• For each one, recorded the optimum network structure for the
given dataset structure (number of fields, types of fields, etc)
• Trained a model to predict the optimum network structure for any
given dataset.
• This predicted network structure & hyper parameters can be
used directly or as a seed for a more intensive network search
Source: BigML - DeepNets https://blog.bigml.com/2017/10/04/deepnets-behind-the-scenes/
#MLSEV
Future Evolution - ML Platforms
46
The Future
Section 7
We are
here
(mostly)
Simplified* AI Technologies Landscape
* and imperfect
Future:

• Knowledge
representation
(symbolic/
Subsymbolic)

• Planning
(Reinforcement
Learning, Agents)

• Reasoning (Logic,
Symbolic)

• Search &
Optimization
(evolutionary/
genetic algos)
48
BigML, IncPrivate and Confidential
BigML Product Progression
5
AutoML, Linear
Regression, Node-
Red, Workflow
Report, Improved
Topic Modeling
Organizations,
Operating
Thresholds, OptiML,
Fusions, Data
Transformations, PCA
Boosted Trees,
ROC Analysis,
Time Series,
DeepNets
Scripts, Libraries,
Executions,
WhizzML, Logistic
Regression, Topic
Models
Association
Discovery,
Correlations,
Samples,
Statistical Tests
Anomaly Detection,
Clusters, Flatline
Evaluations, Batch
Predictions,
Ensembles,
Starbursts
Core ML Workflow:
Source, Dataset,
Model, Prediction
Prototyping and
Beta
201920182017201620152014201320122011
Automating Model Creation, Selection, Operation and Workflows = Making Machine Learning Easier
Reproducibility at the core:
Programmability, Interpretability, Explainability are
essential part of BigML's platform
Sophistication
EaseofUse
WE HAVE BEEN BUILDING A STRONG FOUNDATION TO DEVELOP, DEPLOY AND OPERATE MACHINE-LEARNING BASED APPLICATIONS OF UNPARALLELED QUALITY
49
BigML, IncPrivate and Confidential7
AI/MLMarketMaturity
Automating Workflows for
Model Creation,
Selection, Operation
Extending the Platform to Build and Manage Smarter Predictive Applications End-to-End
Building the BEST End-
to-End Machine
Learning Platform
2020 20301980
BigML's Co-Founder
Participates in first University
Machine Learning
2011
BigML
Founded
BigML Future
EXTENDING THE PLATFORM TO BUILD AND MANAGE SMARTER PREDICTIVE APPLICATIONS END-TO-END
Reasoning
Knowledge
Representation
Planning Optimization
Principles
Machine Learning
ROBUST AI
Doing to Reasoning, Planning, Knowledge Representation
and Optimization what we have done to Machine Learning
and combining them to build Robust AI Applications
Machine Learning
Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterprise
51
1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf
2. ML Systems - Jeff Smith (book)
3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773
4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/
5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/
7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/
citation.cfm?id=2843948.
8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on-
airbnb-9272d3d4739d.
9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management-
platform.
10.Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone.
11.Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018-
facebook.pdf.
12.Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/.
13.Kubeflow, “Kubeflow: Machine Learning Toolkit for Kubernetes,” Github, https://github.com/ kubeflow/.
14.James Kanter and Kalyan Veeramachaneni, “Deep Feature Synthesis: Towards Automating Data Science Endeavors,” 2015, http://www.jmaxkanter.com/static/papers/DSAA_DSM_2015.pdf.
15.Feature Labs, “Featuretools: An Open Source Python Framework for Automated Feature Engineering,” Github, https://github.com/featuretools/featuretools.
16.Frank Hutter, et al, “SMAC,” AutoML Freiburg-Hannover, https://www.automl.org/automated- algorithm-design/algorithm-configuration/smac/
17.Ruben Martinez-Cantin, “BayesOpt: A Toolbox for Bayesian Optimization, Experimental Design and Stochastic Bandits,” Github https://github.com/rmcantin/bayesopt.
18.Hyperopt, “Hyperopt: Distributed Asynchronous Hyperparameter Optimization in Python” Github, Septermber 4, 2011, https://github.com/hyperopt/hyperopt.
19.UCBerkeley RISELab, “Tune: Scalable Hyperparameter Search,” Github https://github.com/ ray-project/ray/tree/master/python/ray/tune.
20.Cade Metz, “Building A.I. That Can Build A.I.,” The New York Times, November 5, 2017, https:// www.nytimes.com/2017/11/05/technology/machine-learning-artificial-intelligence-ai.html
21.Google, “Cloud AutoML,” Google Cloud, https://cloud.google.com/automl/.
22.Neoklis Polyzotis, et al, “Data Management Challenges in Production Machine Learning,” Google, 2017, https://static.googleusercontent.com/media/research.google.com/en//pubs/ archive/46178.pdf.
23.Julien Simon, “Mastering the Mystical Art of Model Deployment,” Medium, July 28, 2018, https:// medium.com/faun/mastering-the-mystical-art-of-model-deployment-c0cafe011175.
24.Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007.
List
of
References

More Related Content

What's hot

MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleMLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleDatabricks
 
MLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in ProductionMLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in ProductionProvectus
 
Apply MLOps at Scale by H&M
Apply MLOps at Scale by H&MApply MLOps at Scale by H&M
Apply MLOps at Scale by H&MDatabricks
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker
 
Machine Learning and the Elastic Stack
Machine Learning and the Elastic StackMachine Learning and the Elastic Stack
Machine Learning and the Elastic StackYann Cluchey
 
Moving to Databricks & Delta
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & DeltaDatabricks
 
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudVertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudMárton Kodok
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
 
How to Build a ML Platform Efficiently Using Open-Source
How to Build a ML Platform Efficiently Using Open-SourceHow to Build a ML Platform Efficiently Using Open-Source
How to Build a ML Platform Efficiently Using Open-SourceDatabricks
 
Data Platform Architecture Principles and Evaluation Criteria
Data Platform Architecture Principles and Evaluation CriteriaData Platform Architecture Principles and Evaluation Criteria
Data Platform Architecture Principles and Evaluation CriteriaScyllaDB
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluentconfluent
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfChris Bingham
 
MLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at ScaleMLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at ScaleDatabricks
 
Architecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleArchitecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleDatabricks
 
How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform
How to Utilize MLflow and Kubernetes to Build an Enterprise ML PlatformHow to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform
How to Utilize MLflow and Kubernetes to Build an Enterprise ML PlatformDatabricks
 
Using MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks
 
Vertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsMárton Kodok
 

What's hot (20)

MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleMLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
 
MLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in ProductionMLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in Production
 
Apply MLOps at Scale by H&M
Apply MLOps at Scale by H&MApply MLOps at Scale by H&M
Apply MLOps at Scale by H&M
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
 
Machine Learning and the Elastic Stack
Machine Learning and the Elastic StackMachine Learning and the Elastic Stack
Machine Learning and the Elastic Stack
 
Moving to Databricks & Delta
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & Delta
 
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudVertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
How to Build a ML Platform Efficiently Using Open-Source
How to Build a ML Platform Efficiently Using Open-SourceHow to Build a ML Platform Efficiently Using Open-Source
How to Build a ML Platform Efficiently Using Open-Source
 
Data Platform Architecture Principles and Evaluation Criteria
Data Platform Architecture Principles and Evaluation CriteriaData Platform Architecture Principles and Evaluation Criteria
Data Platform Architecture Principles and Evaluation Criteria
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
MLOps in action
MLOps in actionMLOps in action
MLOps in action
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
MLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at ScaleMLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at Scale
 
Architecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleArchitecting Agile Data Applications for Scale
Architecting Agile Data Applications for Scale
 
How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform
How to Utilize MLflow and Kubernetes to Build an Enterprise ML PlatformHow to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform
How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform
 
Using MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOps
 
Vertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflows
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 

Similar to Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterprise

Cloud based Machine Learning Platforms, a review - Sagar Khashu
Cloud based Machine Learning Platforms, a review - Sagar KhashuCloud based Machine Learning Platforms, a review - Sagar Khashu
Cloud based Machine Learning Platforms, a review - Sagar KhashuSagar Khashu
 
Emerging engineering issues for building large scale AI systems By Srinivas P...
Emerging engineering issues for building large scale AI systems By Srinivas P...Emerging engineering issues for building large scale AI systems By Srinivas P...
Emerging engineering issues for building large scale AI systems By Srinivas P...Analytics India Magazine
 
Accelerating Machine Learning as a Service with Automated Feature Engineering
Accelerating Machine Learning as a Service with Automated Feature EngineeringAccelerating Machine Learning as a Service with Automated Feature Engineering
Accelerating Machine Learning as a Service with Automated Feature EngineeringCognizant
 
The A-Z of Data: Introduction to MLOps
The A-Z of Data: Introduction to MLOpsThe A-Z of Data: Introduction to MLOps
The A-Z of Data: Introduction to MLOpsDataPhoenix
 
Practical machine learning
Practical machine learningPractical machine learning
Practical machine learningFaizan Javed
 
Sustainable & Composable Generative AI
Sustainable & Composable Generative AISustainable & Composable Generative AI
Sustainable & Composable Generative AIDebmalya Biswas
 
Norman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureNorman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureAgile Impact Conference
 
Norman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureNorman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureAgile Impact
 
BigMLSchool: ML Platforms and AutoML in the Enterprise
BigMLSchool: ML Platforms and AutoML in the EnterpriseBigMLSchool: ML Platforms and AutoML in the Enterprise
BigMLSchool: ML Platforms and AutoML in the EnterpriseBigML, Inc
 
DutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorDutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorBigML, Inc
 
Accelerating Business and Research Through Automation and Artificial Intellig...
Accelerating Business and Research Through Automation and Artificial Intellig...Accelerating Business and Research Through Automation and Artificial Intellig...
Accelerating Business and Research Through Automation and Artificial Intellig...CloudHesive
 
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweekML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweekEd Fernandez
 
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...Matt Stubbs
 
Studying Software Engineering Patterns for Designing Machine Learning Systems
Studying Software Engineering Patterns for Designing Machine Learning SystemsStudying Software Engineering Patterns for Designing Machine Learning Systems
Studying Software Engineering Patterns for Designing Machine Learning SystemsHironori Washizaki
 
Azure Machine Learning 101
Azure Machine Learning 101Azure Machine Learning 101
Azure Machine Learning 101Renato Jovic
 
AutoML - The Future of AI
AutoML - The Future of AIAutoML - The Future of AI
AutoML - The Future of AINing Jiang
 
AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics Ruben Pertusa Lopez
 

Similar to Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterprise (20)

Cloud based Machine Learning Platforms, a review - Sagar Khashu
Cloud based Machine Learning Platforms, a review - Sagar KhashuCloud based Machine Learning Platforms, a review - Sagar Khashu
Cloud based Machine Learning Platforms, a review - Sagar Khashu
 
Technovision
TechnovisionTechnovision
Technovision
 
Emerging engineering issues for building large scale AI systems By Srinivas P...
Emerging engineering issues for building large scale AI systems By Srinivas P...Emerging engineering issues for building large scale AI systems By Srinivas P...
Emerging engineering issues for building large scale AI systems By Srinivas P...
 
Accelerating Machine Learning as a Service with Automated Feature Engineering
Accelerating Machine Learning as a Service with Automated Feature EngineeringAccelerating Machine Learning as a Service with Automated Feature Engineering
Accelerating Machine Learning as a Service with Automated Feature Engineering
 
The A-Z of Data: Introduction to MLOps
The A-Z of Data: Introduction to MLOpsThe A-Z of Data: Introduction to MLOps
The A-Z of Data: Introduction to MLOps
 
Practical machine learning
Practical machine learningPractical machine learning
Practical machine learning
 
Sustainable & Composable Generative AI
Sustainable & Composable Generative AISustainable & Composable Generative AI
Sustainable & Composable Generative AI
 
Norman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureNorman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application Architecture
 
Norman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureNorman Sasono - Incorporating AI/ML into Your Application Architecture
Norman Sasono - Incorporating AI/ML into Your Application Architecture
 
BigMLSchool: ML Platforms and AutoML in the Enterprise
BigMLSchool: ML Platforms and AutoML in the EnterpriseBigMLSchool: ML Platforms and AutoML in the Enterprise
BigMLSchool: ML Platforms and AutoML in the Enterprise
 
DutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorDutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive Sector
 
Introduction to ML.NET
Introduction to ML.NETIntroduction to ML.NET
Introduction to ML.NET
 
Accelerating Business and Research Through Automation and Artificial Intellig...
Accelerating Business and Research Through Automation and Artificial Intellig...Accelerating Business and Research Through Automation and Artificial Intellig...
Accelerating Business and Research Through Automation and Artificial Intellig...
 
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweekML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek
ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek
 
SESE 2021: Where Systems Engineering meets AI/ML
SESE 2021: Where Systems Engineering meets AI/MLSESE 2021: Where Systems Engineering meets AI/ML
SESE 2021: Where Systems Engineering meets AI/ML
 
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
 
Studying Software Engineering Patterns for Designing Machine Learning Systems
Studying Software Engineering Patterns for Designing Machine Learning SystemsStudying Software Engineering Patterns for Designing Machine Learning Systems
Studying Software Engineering Patterns for Designing Machine Learning Systems
 
Azure Machine Learning 101
Azure Machine Learning 101Azure Machine Learning 101
Azure Machine Learning 101
 
AutoML - The Future of AI
AutoML - The Future of AIAutoML - The Future of AI
AutoML - The Future of AI
 
AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics
 

More from Ed Fernandez

Machine Learning & AI - 2022 intro for pre-college students.pdf
Machine Learning & AI - 2022 intro for pre-college students.pdfMachine Learning & AI - 2022 intro for pre-college students.pdf
Machine Learning & AI - 2022 intro for pre-college students.pdfEd Fernandez
 
Solve for X with AI: a VC view of the Machine Learning & AI landscape
Solve for X with AI: a VC view of the Machine Learning & AI landscapeSolve for X with AI: a VC view of the Machine Learning & AI landscape
Solve for X with AI: a VC view of the Machine Learning & AI landscapeEd Fernandez
 
Telco & OTT (Over The Top) the not-new-anymore Kids in the Block
Telco & OTT (Over The Top) the not-new-anymore Kids in the BlockTelco & OTT (Over The Top) the not-new-anymore Kids in the Block
Telco & OTT (Over The Top) the not-new-anymore Kids in the BlockEd Fernandez
 
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?Ed Fernandez
 
WebCongress lima 2014: Home is where your phone connects to WiFi automatically
WebCongress lima 2014: Home is where your phone connects to WiFi automaticallyWebCongress lima 2014: Home is where your phone connects to WiFi automatically
WebCongress lima 2014: Home is where your phone connects to WiFi automaticallyEd Fernandez
 
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...WebCongress US 2013 why does superman wear his underwear outside his pants?. ...
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...Ed Fernandez
 
TEDx Raval 3 de mar 2012 @efernandez 15min short web
TEDx Raval 3 de mar 2012 @efernandez 15min short webTEDx Raval 3 de mar 2012 @efernandez 15min short web
TEDx Raval 3 de mar 2012 @efernandez 15min short webEd Fernandez
 
Eduardo fernandez #bif11 keynote
Eduardo fernandez #bif11 keynoteEduardo fernandez #bif11 keynote
Eduardo fernandez #bif11 keynoteEd Fernandez
 
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...Ed Fernandez
 
Argumentation The Art Of Persuasion (Colegio De Abogados)
Argumentation   The Art Of Persuasion (Colegio De Abogados)Argumentation   The Art Of Persuasion (Colegio De Abogados)
Argumentation The Art Of Persuasion (Colegio De Abogados)Ed Fernandez
 
M Life 2009 Modern Government And Mobility Eduardo Fernandez Rojo
M Life 2009 Modern Government And Mobility   Eduardo Fernandez RojoM Life 2009 Modern Government And Mobility   Eduardo Fernandez Rojo
M Life 2009 Modern Government And Mobility Eduardo Fernandez RojoEd Fernandez
 
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short Def
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short DefIct Mobile Summit 2009 Eduardo Fernandez Plenary Session Short Def
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short DefEd Fernandez
 

More from Ed Fernandez (12)

Machine Learning & AI - 2022 intro for pre-college students.pdf
Machine Learning & AI - 2022 intro for pre-college students.pdfMachine Learning & AI - 2022 intro for pre-college students.pdf
Machine Learning & AI - 2022 intro for pre-college students.pdf
 
Solve for X with AI: a VC view of the Machine Learning & AI landscape
Solve for X with AI: a VC view of the Machine Learning & AI landscapeSolve for X with AI: a VC view of the Machine Learning & AI landscape
Solve for X with AI: a VC view of the Machine Learning & AI landscape
 
Telco & OTT (Over The Top) the not-new-anymore Kids in the Block
Telco & OTT (Over The Top) the not-new-anymore Kids in the BlockTelco & OTT (Over The Top) the not-new-anymore Kids in the Block
Telco & OTT (Over The Top) the not-new-anymore Kids in the Block
 
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?
WebCongress panama 2014 - War Is Over - What's Next After the Smartphone Wars?
 
WebCongress lima 2014: Home is where your phone connects to WiFi automatically
WebCongress lima 2014: Home is where your phone connects to WiFi automaticallyWebCongress lima 2014: Home is where your phone connects to WiFi automatically
WebCongress lima 2014: Home is where your phone connects to WiFi automatically
 
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...WebCongress US 2013 why does superman wear his underwear outside his pants?. ...
WebCongress US 2013 why does superman wear his underwear outside his pants?. ...
 
TEDx Raval 3 de mar 2012 @efernandez 15min short web
TEDx Raval 3 de mar 2012 @efernandez 15min short webTEDx Raval 3 de mar 2012 @efernandez 15min short web
TEDx Raval 3 de mar 2012 @efernandez 15min short web
 
Eduardo fernandez #bif11 keynote
Eduardo fernandez #bif11 keynoteEduardo fernandez #bif11 keynote
Eduardo fernandez #bif11 keynote
 
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...
UIMP Visiones Internacionales desde España y Nuevos escenarios estrategicos s...
 
Argumentation The Art Of Persuasion (Colegio De Abogados)
Argumentation   The Art Of Persuasion (Colegio De Abogados)Argumentation   The Art Of Persuasion (Colegio De Abogados)
Argumentation The Art Of Persuasion (Colegio De Abogados)
 
M Life 2009 Modern Government And Mobility Eduardo Fernandez Rojo
M Life 2009 Modern Government And Mobility   Eduardo Fernandez RojoM Life 2009 Modern Government And Mobility   Eduardo Fernandez Rojo
M Life 2009 Modern Government And Mobility Eduardo Fernandez Rojo
 
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short Def
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short DefIct Mobile Summit 2009 Eduardo Fernandez Plenary Session Short Def
Ict Mobile Summit 2009 Eduardo Fernandez Plenary Session Short Def
 

Recently uploaded

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Recently uploaded (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 

Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterprise

  • 1. 2 Machine Learning Platformization & AutoML Adopting ML at scale in the Enterprise Ed Fernandez @efernandez
  • 2. 3 Disclaimer: The term AI (Artificial Intelligence) appears several times throughout these slides in several references and 3rd party content In the context of this presentation it refers specifically to the ability to build machine learning driven applications which ultimately automate and/or optimize business processes and DOES NOT refer to true or strong Artificial Intelligence in the formal sense, which is not likely to happen for decades to come (emphasis from author) 1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf 2. ML Systems - Jeff Smith (book) 3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773 4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/ 5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/ 7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/citation.cfm?id=2843948. 8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on-airbnb-9272d3d4739d. 9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management-platform. 10. Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone. 11. Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018-facebook.pdf. 12. Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/. 13. Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007. 14. BigML Documentation https://bigml.com/documentation/ 15. Domain Specific Language for ML Workflows Automation - WhizzML - BigML https://bigml.com/whatsnew/whizzml#whizzml-automating-machine-learning 16. Domain Specific Language for Feature Engineering - Flatline https://github.com/bigmlcom/flatline 17. AutoML - OptiML https://bigml.com/api/optimls References (Partial List):
  • 3. #MLSEV [1] Definitions & Context •Machine Learning Platforms, Definitions •ML models & apps as first class assets in the Enterprise •Workflow of an ML application •ML Algorithms, overview •Architecture of a ML platform •Update on the Hype cycle for ML & predictive apps [2] Adopting ML at Scale •The Problem with Machine Learning - Scaling ML in the Enterprise •Technical Debt in ML systems •How many models are too many models •The need for ML platforms [3] The Market for ML Platforms •ML platform Market References - from early adopters to mainstream •Custom Build vs Buy: ROI & Technical Debt •ML Platforms - Vendor Landscape 4 Summary [7] Future Evolution for ML Platforms Appendix I: Practical Recommendations for ML onboarding in the Enterprise Appendix II: List of References & Additional Resources [4] Custom Built ML Platforms • ML platform Market References - a closer look Facebook - FBlearner Uber - Michelangelo AirBnB - BigHead • ML Platformization Going Mainstream: The Great Enterprise Pivot [5] From DevOps to MLOps • DevOps <> ModelOps • The ML platform driven Organization • Leadership & Accountability (labour division) [6] Automated ML - AutoML • Scaling ML - Rapid Prototyping & AutoML: • Definition, Rationale • Vendor Comparison • AutoML - OptiML: Use Cases
  • 4. #MLSEV • Machine Learning Platforms, Definitions • ML use cases & apps as first class assets in the Enterprise • Workflow of an ML application • ML Algorithms, overview • Architecture of an ML platform • Update on the Hype cycle for ML & predictive apps 5 Definitions & Context Section 1
  • 5. 6 The ML platform offers advanced functionality essential for building ML solutions (primarily predictive and prescriptive models). The platform supports the incorporation of these solutions into business processes, surrounding infrastructure, products and applications. It supports variously skilled data scientists (and other stakeholders i.e ML engineers, Data Analysts & Business Analysts and experts) in multiple tasks across the data and analytics pipeline, including all of the following areas: • Data ingestion 
 • Data preparation & Transformation
 • Data exploration & Visualization
 • Feature engineering • Model Selection, Evaluation & testing (AutoML)
 • Deployment 
 • Monitoring 
 • Maintenance 
 • Collaboration Machine Learning Platforms A formal definition The workflow of a machine learning project. Defining a problem, prototyping a solution, productionizing the solution and measuring the impact of the solution is the core workflow. The loops throughout the workflow represent the many iterations of feedback gathering needed to perfect the solution and complete the project. Adapted from Gartner DSML Data Science and Machine Learning Platforms report, February 2020 - ID G00385005
  • 6. Internal & External AI assets: ML modeling, heuristics AI assets: ML platform AI assets: People skills/expertise ML Adoption cross-function Enterprise Roadmap for AI & ML ML models as first-class enterprise asset
  • 7. SUPERVISED UNSUPERVISED DATA Requires “labelled” data Does not require “labelled” data GOAL Goal is to predict the label often called the objective (churn, sales predictions, etc). Goal is “structure discovery”, with algorithms focused on type of relation (clustering, etc.) EVALUATION Predictions can be compared to real labels Each algorithm has it’s own quality measures ALGORITHMS ML Algorithms 8 CLUSTER ANOMALY TOPIC MODEL ASSOCIATIONTREE MODEL ENSEMBLE DEEPNETLOGISTIC REGRESSION TIME SERIES CLASSIFICATION / REGRESSION OPTIML
  • 8. 9 Deep Learning: Specific Use Cases ANN CNN & RNN Bayesian NN (traditional) Machine Learning: Workhorse algorithms Linear & Logistic Regression Decision Trees & Random Forest Ensembles source: Kaggle · The State of Data Science & ML 2019 · https://www.kaggle.com/kaggle-survey-2019 Machine Learning Adoption ML Algorithms in practice
  • 9. BigML, Inc Where are my models? 10 Architecture of a ML Platform - MLaaS - BigML • Models are stored in the BigML server, in the cloud. • Private and On premises clouds are also available. • API first: every execution (model, dataset, evaluation, automation script) is an immutable resource that can be managed programmatically. • Resources are encoded in JSON. are easy to integrate and export to other applications and workflows API-first, auto-scalable, auto-deployable distributed architecture for Machine Learning
  • 11. Emerging Technology hype cycle: Machine Learning The Great Enterprise Pivot We are here ~2 years to mainstream
  • 12. Adoption Cycle: Machine Learning Custom Built vs Buy, crossing the chasm source: adapted from BigML Inc materials · http://bigml.com We are here • Open Source • Custom Built vs Buy • Fragmented • Proprietary • Buy vs Build • Consolidated
  • 13. #MLSEV The Problem with Machine Learning - Adopting ML at Scale in the Enterprise Technical Debt in ML systems How many models are too many models The need for ML platforms 14 Adopting ML at Scale Section 2
  • 14. “ The problem with Machine Learning Adopting Machine Learning at Scale in the Enterprise It is time to bring the AI exploration era to the next stage of production - enabling sustainable, industrial- grade AI systems within the IT and cultural fabric. Gartner “Artificial Intelligence Primer for 2020” Erick Brethenoux, 24 January 2020
  • 15. 16 The problem with Machine Learning source: Kaggle · The State of Data Science & ML 2019 · https://www.kaggle.com/kaggle-survey-2019 From prototyping to production
  • 16. 17 D. Sculley et al., Google, NIPS 2015 Technical Debt in Machine Learning Model Drifting - Data Lifecycle
  • 17. 18 NIPS: Hidden technical debt in ML https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf Dealing with Complexity Infrastructure & Fragmentation
  • 18. 19 How many ML models are too many models Facebook ML platform (a.k.a FBlearner): +1Mn ML models trained +6 Mn predictions/sec 25% of engineering team using it Source: ModelOps IBM research Waldemar Hummer et al http://hummer.io/docs/2019-ic2e-modelops.pdf
  • 19. 20 Source: David Talby CTO, Pacific AI - Strata Conference https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68616 Increasing number of models & complexity ML Use Cases
  • 20. 21 Increasing number of models & complexity Uber Facebook Twitter Linkedin SO PUT THE RIGHT ML PLATFORM IN PLACE THESE COMPANIES DID ALREADY (Custom Built) •e-commerce •online/real time transaccions •consumer C2C services •Predictions driven by volume (millions) & models •long term trends & patterns •B2B & Government services •consumer C2C services •Predictions driven by quality & •rules based knowledge AirBnB Lyft Netflix Spotify GE AT&T eBay Amazon
  • 21. #MLSEV ML platform Market References - from early adopters to mainstream Custom Build vs Buy: ROI & Technical Debt ML Platforms - Vendor Landscape 22 The Market for ML Platforms Section 3
  • 22. Amazon Jeff Bezos’ letter to Amazon shareholders - May, 2017 “Machine learning and AI is a horizontal enabling layer. It will empower and improve every business, every government organization, every philanthropy — basically there’s no institution in the world that cannot be improved with machine learning” . Jeff Bezos
  • 23. Machine Learning Platforms An Infrastructure & Service layer to drive ML at scale in the enterprise Facebook FBlearner May 9, 2016 https://code.fb.com/core-data/ introducing-fblearner-flow-facebook-s- ai-backbone/ Google TFX Tensorflow Aug 13, 2017 https://www.tensorflow.org/tfx/ https://dl.acm.org/ft_gateway.cfm? id=3098021&ftid=1899117&dwn=1&CF ID=81485403&CFTOKEN=79729647b 2ac491f-EAC34BCC-93F2-A3C5- BE9311C722468452 Netflix Notebook Data Platform Aug 16, 2018 https://medium.com/netflix-techblog/ notebook-innovation-591ee3221233 Uber Michelangelo Sept 5, 2017 https://eng.uber.com/michelangelo/ Twitter Cortex Sept, 2015 https://cortex.twitter.com/en.html https://blog.twitter.com/engineering/ en_us/topics/insights/2018/ml- workflows.html Magic Pony acquisition - 2016: https://www.bernardmarr.com/ default.asp?contentID=1373 AirBnB BigHead Feb, 2018 https://databricks.com/session/ bighead-airbnbs-end-to-end-machine- learning-platform LinkedIN Pro-ML Oct, 2018 https://engineering.linkedin.com/blog/ 2018/10/an-introduction-to-ai-at- linkedin
  • 25. Machine Learning Platforms eBay Krylov Dec 17, 2019 https://tech.ebayinc.com/engineering/ ebays-transformation-to-a-modern-ai- platform/ Lyft Flyte Jan 20, 2020 https://eng.lyft.com/introducing-flyte- cloud-native-machine-learning-and- data-processing-platform- fb2bb3046a59 AT&T Acumos Oct 30, 2017 https://medium.com/netflix-techblog/ notebook-innovation-591ee3221233 Spotify Spotify ML platform Dec 13, 2019 https://labs.spotify.com/2019/12/13/the- winding-road-to-better-machine- learning-infrastructure-through- tensorflow-extended-and-kubeflow/ Delta Airlines (licensed) Jan 8, 2020 https://www.aviationtoday.com/ 2020/01/08/delta-develops-ai-tool- address-weather-disruption-improve- flight-operations/ GE Predix (customer IoT platform) Feb, 2018 https://www.ge.com/digital/sites/ default/files/download_assets/Predix- The-Industrial-Internet-Platform- Brief.pdf KT Telecom Neuroflow Jan, 2018 https://disruptive.asia/kt-ai-platform- internal-use/ An Infrastructure & Service layer to drive ML at scale in the enterprise
  • 26. Machine Learning Platforms Build vs Buy The “custom build” approach, while highly customized to the needs of the organization, is expensive, requires time and strong engineering talent and teams to develop and maintain it The “buy” option often requires adapting to a given vendor’s approach but demands less time and expertise and provides continued access to innovations Ultimately, it’s a business case decision (ROI calculator next slide) Partial list of ML platform licensees (courtesy of BigML Inc) Most enterprises will ultimately implement their ML platforms from commercial or cloud-delivered software, along with custom integration and custom- coded modules tailored to their specific needs
  • 27. 28 ML Platform Build vs Buy ROI Source: Dataiku DS ROI toolkit https://pages.dataiku.com/data-science-roi-toolkit
  • 28. 29 MACHINE LEARNING AS A SERVICE MACHINE LEARNING PLATFORM & SOFTWARE https://www.crisp-research.com/vendor-universe/machine-learning/#fndtn-mlaas Machine Learning Platforms Vendor Landscape MLaaS: Machine Learning as a Service & On Premise
  • 29. 30 ML Platformization Going Mainstream Buy vs Build Partial list of ML platform public customer references: HG Insights (BigML Inc, Dataiku & H2O.ai https://discovery.hgdata.com/product/bigml)
  • 30. #MLSEV ML platform Market References - a closer look •Facebook - FBlearner •Uber - Michelangelo •AirBnB - BigHead ML Platformization Going Mainstream: The Great Enterprise Pivot 31 Custom Built ML Platforms Section 4
  • 31. Facebook FBlearner Flow: Facebook’s ML platform for internal use - May, 2016 25% of engineering team using it +1Mn ML models trained +6 Mn predictions/sec ML at scale: Reusability Parallelization Simplicity Automation Rapid prototyping & experimentation
  • 32. Facebook FBlearner Flow: Facebook’s ML platform for internal use - May, 2016 Eliminating manual work for experimentation Engineers can spend more time on feature engineering which in turn produce greater accuracy improvements “
  • 33. Uber Michelangelo: Uber’s MLaaS platform for internal use - Sept, 2017 end-to-end ML workflow: • manage data • train • evaluate • deploy models • make and monitor predictions. Supports traditional ML models, time series forecasting, and deep learning.
  • 34. 35 Uber Michelangelo: Uber’s MLaaS platform for internal use - Sept, 2017
  • 35. AirBnB Bighead - Feb, 2018 Airbnb’s internal ML platform is called Bighead. Bighead is an end-to-end platform for building and deploying ML models that aims to make the machine learning process at Airbnb seamless, versatile, consistent, and scalable. It is built in Python and relies on open source technology like Docker, Jupyter, Spark, Kubernetes, and more. These open source components are customized and integrated for Airbnb’s specific needs. Like much of Airbnb’s technology infrastructure, Bighead runs in AWS. The platform was supported by an ML infrastructure team of 11 engineers and one product manager. In the fall of 2018, Airbnb announced its plans to open source parts of Bighead and Zipline in early 2019, but this hasn’t yet materialized.
  • 36. The Great Pivot - ML at scale Systems of Intelligence/ML drive efficiencies (1st), competitive advantages (2nd) & next defensible business models ultimately • Most large technology companies are reconfiguring themselves around ML. • Google was (arguably) the first company to move, followed by Microsoft, Facebook, Amazon, Apple and IBM. • 2nd tier corporations following suit: GE, Uber, even carriers as AT&T • Not only a US phenomena - Alibaba, Baidu chief Robin Li said in an internal memo that Baidu’s strategic future relies on AI • Ultimately all global players will need to re-tool their processes adopting a ML driven approach.h/t Jerry Chen - Greylock Partners https://news.greylock.com/the-new-moats-53f61aeac2d9
  • 37. #MLSEV Scaling ML - Rapid Prototyping & AutoML: Definition, Rationale Vendor Comparison AutoML - OptiML: Use Cases 38 Automated ML - AutoML Section 6
  • 38. “All Models are wrong, but some are useful”
  • 39. AutoML Automated Machine Learning 40 Problem Formulation Data Acquisition Feature Engineering Modeling and Evaluations Predictions Measure Results Data Transformations 5% 80% • Data tasks, most consuming - Semi automated. • Feature Engineering is key to model performance - semi automated 10% • Goal definition - Human driven 5% • AutoML enables fast modeling/prototyping - Automated • Automated
  • 40. 41 Enable knowledge workers (e.g., analysts, developers) to build stable and insightful models quickly Scale the number of predictive use cases in collaboration with non-technical peers through quick prototyping. Best AutoML approaches rely on automation of parts of the Machine Learning process (e.g., hyper-parameter tuning) without limiting the practitioners’ ability control customization. GDPR, data privacy, interpretability and prediction explanations became critical concerns when deploying AutoML AutoML Automated Machine Learning
  • 42. 43 AutoML Trade off in Model/Algorithm Selection • Simple (Logistic Reg) vs Complex (Deepnets, ANNs) • Weak and Fast vs. Slow and Robust • Interpretability vs. Representability • Confidence vs. Performance • Biased vs. Data-hungry
  • 43. 44 AutoML DATAROBOT H2O BigML Data Preparation • Encoded categorical variables (one-hot); Text n- grams; Missing values imputing; Discretization (bins) • limited manual transformations • Max. of 10 classes in the objective* •Encoded categorical variables (one-hot); Missing values handling; Date-time fields expansion; Bulk interactions transformers; SVD numeric transformer; CV target encoding; Cluster distance transformer; Time lag •Automatic feature engineering possible when using AutoDL • Encoded categorical variables (one-hot); Text analysis; Missing values handling; Date-time fields expansion • Automatic Recursive Feature Selection & Feature Engineering • Multiple flexible manual transformations • Max of 1,000 classes in the objective Optimization Undisclosed optimization technique (“expert data scientists preset hyperparameter search space for models*) Random Stacking (a combination of random grid search and stacked ensembles, plus early stopping) Bayesian Parameter Optimization (SMAC — Sequential Model-based Algorithm Configuration) & DNN Metalearning Models •Open-source libraries: scikit-learn, R, H2O, Tensorflow (not CNN or RNN), Spark, XGBoost, DMTK, and Vowpal Wabbit 
 •They also “blend” multiple models during the optimization process. •GBMs, Random Forests, XGBoost, deep neural nets, and extreme random forests •· Stacks of models can be learned. Best of family stacks adopt the top model type from each of the main algorithms. •Decision trees, random decision forests, boosting, logistic regression, deep neural networks 
 •Customizable model ensembles with Fusions leveraging the individually optimized models for different classification, regression algorithms. Speed It tests 30-40 different modeling approaches and takes ~20 min. Default time limit for AutoML is 1 hour. Can use GPU or CPU. Can specify settings for accuracy, time, and interpretability. It tests 128 different modeling approaches (creating more than 500 resources) and takes ~30 min. Model Visualizations & Interpretability • Limited model visualizations • Feature importance for models • Predictions explainability • Dashboard: A single page with a global interpretable model explanations plot, a feature importance plot, a decision tree plot, and a partial dependence plot. • A machine learning interpretation tool (MLI) that includes a KLIME or LIME-SUP graph. • Multiple model visualizations to analyze the impact of the variables on predictions: sunburst, decision tree, partial dependence plots, line chart (LR) • Feature importance for models • Predictions explainability Model Evaluations • Confusion matrix
 • ROC curve (only for binary classification)
 • Lift curve (only for binary classification)
 • Side-by-side evaluations comparison
 • Trade-off between complexity vs. performance • Models are ranked by cross-validation 
 AUC by default. • Return leaderboard sortable by deviance (mean residual deviance), logloss, MSE, RMSE, MAE, RMSLE, mean per class error • Confusion matrix
 • ROC curve
 • Precision-Recall curve
 • Gain curve
 • Lift curve
 • Multiple evaluations comparison chart Programmability & Deployability • Models can be used and created via API • Export models
 • Cloud, VPC or on-premises • H2O allows you to convert the models you have built to either a Plain Old Java Object (POJO) or a Model ObJect, Optimized (MOJO). • H2O-generated MOJO and POJO models are ieasily embeddable in Java environments • Models can be used and created via API • Export models
 • Cloud, VPC or on-premises Source: Public Resources, Vendor Docs, BigML Analysis Metalearning!
  • 44. 45 AutoML - Metalearning Automatic Network Hyperparameters Selection - DNNs (DeepNets) We trained 296,748 deep neural networks so you don’t have to! • 296,748+ deep neural networks trained on 50 datasets • For each one, recorded the optimum network structure for the given dataset structure (number of fields, types of fields, etc) • Trained a model to predict the optimum network structure for any given dataset. • This predicted network structure & hyper parameters can be used directly or as a seed for a more intensive network search Source: BigML - DeepNets https://blog.bigml.com/2017/10/04/deepnets-behind-the-scenes/
  • 45. #MLSEV Future Evolution - ML Platforms 46 The Future Section 7
  • 46. We are here (mostly) Simplified* AI Technologies Landscape * and imperfect Future: • Knowledge representation (symbolic/ Subsymbolic) • Planning (Reinforcement Learning, Agents) • Reasoning (Logic, Symbolic) • Search & Optimization (evolutionary/ genetic algos)
  • 47. 48 BigML, IncPrivate and Confidential BigML Product Progression 5 AutoML, Linear Regression, Node- Red, Workflow Report, Improved Topic Modeling Organizations, Operating Thresholds, OptiML, Fusions, Data Transformations, PCA Boosted Trees, ROC Analysis, Time Series, DeepNets Scripts, Libraries, Executions, WhizzML, Logistic Regression, Topic Models Association Discovery, Correlations, Samples, Statistical Tests Anomaly Detection, Clusters, Flatline Evaluations, Batch Predictions, Ensembles, Starbursts Core ML Workflow: Source, Dataset, Model, Prediction Prototyping and Beta 201920182017201620152014201320122011 Automating Model Creation, Selection, Operation and Workflows = Making Machine Learning Easier Reproducibility at the core: Programmability, Interpretability, Explainability are essential part of BigML's platform Sophistication EaseofUse WE HAVE BEEN BUILDING A STRONG FOUNDATION TO DEVELOP, DEPLOY AND OPERATE MACHINE-LEARNING BASED APPLICATIONS OF UNPARALLELED QUALITY
  • 48. 49 BigML, IncPrivate and Confidential7 AI/MLMarketMaturity Automating Workflows for Model Creation, Selection, Operation Extending the Platform to Build and Manage Smarter Predictive Applications End-to-End Building the BEST End- to-End Machine Learning Platform 2020 20301980 BigML's Co-Founder Participates in first University Machine Learning 2011 BigML Founded BigML Future EXTENDING THE PLATFORM TO BUILD AND MANAGE SMARTER PREDICTIVE APPLICATIONS END-TO-END Reasoning Knowledge Representation Planning Optimization Principles Machine Learning ROBUST AI Doing to Reasoning, Planning, Knowledge Representation and Optimization what we have done to Machine Learning and combining them to build Robust AI Applications Machine Learning
  • 50. 51 1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf 2. ML Systems - Jeff Smith (book) 3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773 4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/ 5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/ 7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/ citation.cfm?id=2843948. 8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on- airbnb-9272d3d4739d. 9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management- platform. 10.Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone. 11.Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018- facebook.pdf. 12.Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/. 13.Kubeflow, “Kubeflow: Machine Learning Toolkit for Kubernetes,” Github, https://github.com/ kubeflow/. 14.James Kanter and Kalyan Veeramachaneni, “Deep Feature Synthesis: Towards Automating Data Science Endeavors,” 2015, http://www.jmaxkanter.com/static/papers/DSAA_DSM_2015.pdf. 15.Feature Labs, “Featuretools: An Open Source Python Framework for Automated Feature Engineering,” Github, https://github.com/featuretools/featuretools. 16.Frank Hutter, et al, “SMAC,” AutoML Freiburg-Hannover, https://www.automl.org/automated- algorithm-design/algorithm-configuration/smac/ 17.Ruben Martinez-Cantin, “BayesOpt: A Toolbox for Bayesian Optimization, Experimental Design and Stochastic Bandits,” Github https://github.com/rmcantin/bayesopt. 18.Hyperopt, “Hyperopt: Distributed Asynchronous Hyperparameter Optimization in Python” Github, Septermber 4, 2011, https://github.com/hyperopt/hyperopt. 19.UCBerkeley RISELab, “Tune: Scalable Hyperparameter Search,” Github https://github.com/ ray-project/ray/tree/master/python/ray/tune. 20.Cade Metz, “Building A.I. That Can Build A.I.,” The New York Times, November 5, 2017, https:// www.nytimes.com/2017/11/05/technology/machine-learning-artificial-intelligence-ai.html 21.Google, “Cloud AutoML,” Google Cloud, https://cloud.google.com/automl/. 22.Neoklis Polyzotis, et al, “Data Management Challenges in Production Machine Learning,” Google, 2017, https://static.googleusercontent.com/media/research.google.com/en//pubs/ archive/46178.pdf. 23.Julien Simon, “Mastering the Mystical Art of Model Deployment,” Medium, July 28, 2018, https:// medium.com/faun/mastering-the-mystical-art-of-model-deployment-c0cafe011175. 24.Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007. List of References