Adoption of ML at scale in the Enterprise, Machine Learning Platforms & AutoML
[1] Definitions & Context
• Machine Learning Platforms, Definitions
• ML models & apps as first class assets in the Enterprise
• Workflow of an ML application
• ML Algorithms, overview
• Architecture of a ML platform
• Update on the Hype cycle for ML & predictive apps
[2] Adopting ML at Scale
• The Problem with Machine Learning - Scaling ML in the
Enterprise
• Technical Debt in ML systems
• How many models are too many models
• The need for ML platforms
[3] The Market for ML Platforms
• ML platform Market References - from early adopters to
mainstream
• Custom Build vs Buy: ROI & Technical Debt
• ML Platforms - Vendor Landscape
[4] Custom Built ML Platforms
• ML platform Market References - a closer look
Facebook - FBlearner
Uber - Michelangelo
AirBnB - BigHead
• ML Platformization Going Mainstream: The Great Enterprise Pivot
[5] From DevOps to MLOps
• DevOps <> ModelOps
• The ML platform driven Organization
• Leadership & Accountability (labour division)
[6] Automated ML - AutoML
• Scaling ML - Rapid Prototyping & AutoML:
• Definition, Rationale
• Vendor Comparison
• AutoML - OptiML: Use Cases
[7] Future Evolution for ML Platforms
Appendix I: Practical Recommendations for ML onboarding in the Enterprise
Appendix II: List of References & Additional Resources
2. 3
Disclaimer:
The term AI (Artificial Intelligence) appears several times throughout these slides in several references and 3rd party content
In the context of this presentation it refers specifically to the ability to build machine learning driven applications which
ultimately automate and/or optimize business processes and DOES NOT refer to true or strong Artificial Intelligence in the
formal sense, which is not likely to happen for decades to come (emphasis from author)
1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf
2. ML Systems - Jeff Smith (book)
3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773
4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/
5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/
7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/citation.cfm?id=2843948.
8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on-airbnb-9272d3d4739d.
9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management-platform.
10. Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone.
11. Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018-facebook.pdf.
12. Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/.
13. Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007.
14. BigML Documentation https://bigml.com/documentation/
15. Domain Specific Language for ML Workflows Automation - WhizzML - BigML https://bigml.com/whatsnew/whizzml#whizzml-automating-machine-learning
16. Domain Specific Language for Feature Engineering - Flatline https://github.com/bigmlcom/flatline
17. AutoML - OptiML https://bigml.com/api/optimls
References (Partial List):
3. #MLSEV
[1] Definitions & Context
•Machine Learning Platforms, Definitions
•ML models & apps as first class assets in the Enterprise
•Workflow of an ML application
•ML Algorithms, overview
•Architecture of a ML platform
•Update on the Hype cycle for ML & predictive apps
[2] Adopting ML at Scale
•The Problem with Machine Learning - Scaling ML in the
Enterprise
•Technical Debt in ML systems
•How many models are too many models
•The need for ML platforms
[3] The Market for ML Platforms
•ML platform Market References - from early adopters to
mainstream
•Custom Build vs Buy: ROI & Technical Debt
•ML Platforms - Vendor Landscape
4
Summary
[7] Future Evolution for ML Platforms
Appendix I: Practical Recommendations for ML onboarding in the
Enterprise
Appendix II: List of References & Additional Resources
[4] Custom Built ML Platforms
• ML platform Market References - a closer look
Facebook - FBlearner
Uber - Michelangelo
AirBnB - BigHead
• ML Platformization Going Mainstream: The Great Enterprise Pivot
[5] From DevOps to MLOps
• DevOps <> ModelOps
• The ML platform driven Organization
• Leadership & Accountability (labour division)
[6] Automated ML - AutoML
• Scaling ML - Rapid Prototyping & AutoML:
• Definition, Rationale
• Vendor Comparison
• AutoML - OptiML: Use Cases
4. #MLSEV
•
Machine Learning Platforms, Definitions
•
ML use cases & apps as first class assets in the Enterprise
•
Workflow of an ML application
•
ML Algorithms, overview
•
Architecture of an ML platform
•
Update on the Hype cycle for ML & predictive apps
5
Definitions & Context
Section 1
5. 6
The ML platform offers advanced functionality essential for building ML solutions (primarily predictive and prescriptive models).
The platform supports the incorporation of these solutions into business processes, surrounding infrastructure, products and applications.
It supports variously skilled data scientists (and other stakeholders i.e ML engineers, Data Analysts & Business Analysts and experts) in
multiple tasks across the data and analytics pipeline, including all of the following areas:
• Data ingestion
• Data preparation & Transformation
• Data exploration & Visualization
• Feature engineering
• Model Selection, Evaluation & testing (AutoML)
• Deployment
• Monitoring
• Maintenance
• Collaboration
Machine Learning Platforms
A formal definition
The workflow of a machine learning project. Defining a problem, prototyping a solution, productionizing the solution and measuring the impact of
the solution is the core workflow. The loops throughout the workflow represent the many iterations of feedback gathering needed to perfect the
solution and complete the project.
Adapted from Gartner DSML Data Science and Machine Learning Platforms report, February 2020 - ID G00385005
6. Internal &
External
AI assets:
ML modeling,
heuristics
AI assets:
ML platform
AI assets:
People
skills/expertise
ML Adoption
cross-function
Enterprise Roadmap for AI & ML
ML models as first-class enterprise asset
7. SUPERVISED UNSUPERVISED
DATA Requires “labelled” data Does not require “labelled” data
GOAL
Goal is to predict the label often called the objective
(churn, sales predictions, etc).
Goal is “structure discovery”, with
algorithms focused on type of relation
(clustering, etc.)
EVALUATION Predictions can be compared to real labels
Each algorithm has it’s own quality
measures
ALGORITHMS
ML Algorithms
8
CLUSTER ANOMALY
TOPIC
MODEL
ASSOCIATIONTREE
MODEL
ENSEMBLE DEEPNETLOGISTIC
REGRESSION
TIME SERIES
CLASSIFICATION / REGRESSION
OPTIML
8. 9
Deep Learning:
Specific Use
Cases
ANN
CNN & RNN
Bayesian NN
(traditional)
Machine Learning:
Workhorse
algorithms
Linear & Logistic
Regression
Decision Trees &
Random Forest
Ensembles
source: Kaggle · The State of Data Science
& ML 2019 ·
https://www.kaggle.com/kaggle-survey-2019
Machine Learning Adoption
ML Algorithms in practice
9. BigML, Inc
Where are my models?
10
Architecture of a ML Platform - MLaaS - BigML
• Models are stored in the BigML server, in the cloud.
• Private and On premises clouds are also available.
• API first: every execution (model, dataset,
evaluation, automation script) is an immutable
resource that can be managed programmatically.
• Resources are encoded in JSON. are easy to
integrate and export to other applications and
workflows
API-first, auto-scalable, auto-deployable
distributed architecture for Machine Learning
11. Emerging Technology hype cycle: Machine Learning
The Great Enterprise Pivot
We are here
~2 years to
mainstream
12. Adoption Cycle: Machine Learning
Custom Built vs Buy, crossing the chasm
source: adapted from BigML Inc materials · http://bigml.com
We are here
• Open
Source
• Custom Built
vs Buy
• Fragmented
• Proprietary
• Buy vs Build
• Consolidated
13. #MLSEV
The Problem with Machine Learning - Adopting ML at Scale in the
Enterprise
Technical Debt in ML systems
How many models are too many models
The need for ML platforms
14
Adopting ML at Scale
Section 2
14. “
The problem with Machine Learning
Adopting Machine Learning at Scale in the Enterprise
It is time to bring the AI exploration
era to the next stage of production -
enabling sustainable, industrial-
grade AI systems within the IT and
cultural fabric.
Gartner
“Artificial Intelligence Primer for 2020” Erick Brethenoux, 24 January 2020
15. 16
The problem with Machine Learning
source: Kaggle · The State of Data Science &
ML 2019 ·
https://www.kaggle.com/kaggle-survey-2019
From prototyping to production
16. 17
D. Sculley et al., Google, NIPS 2015
Technical Debt in Machine Learning
Model Drifting - Data Lifecycle
17. 18
NIPS: Hidden technical debt in ML
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
Dealing with Complexity
Infrastructure & Fragmentation
18. 19
How many ML models are too many models
Facebook ML platform (a.k.a FBlearner):
+1Mn ML models trained
+6 Mn predictions/sec
25% of engineering team using it
Source: ModelOps IBM research Waldemar Hummer et al http://hummer.io/docs/2019-ic2e-modelops.pdf
19. 20
Source: David Talby CTO, Pacific AI - Strata Conference
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68616
Increasing number of models & complexity
ML Use Cases
20. 21
Increasing number of models & complexity
Uber
Facebook
Twitter
Linkedin
SO PUT THE RIGHT ML PLATFORM IN PLACE
THESE COMPANIES DID ALREADY (Custom Built)
•e-commerce
•online/real time
transaccions
•consumer C2C services
•Predictions driven by
volume (millions) & models
•long term trends &
patterns
•B2B & Government
services
•consumer C2C services
•Predictions driven by
quality &
•rules based knowledge
AirBnB
Lyft
Netflix
Spotify
GE
AT&T
eBay
Amazon
21. #MLSEV
ML platform Market References - from early adopters to mainstream
Custom Build vs Buy: ROI & Technical Debt
ML Platforms - Vendor Landscape
22
The Market for ML Platforms
Section 3
22. Amazon
Jeff Bezos’ letter to Amazon shareholders - May, 2017
“Machine learning and AI is a horizontal
enabling layer. It will empower and improve
every business, every government
organization, every philanthropy — basically
there’s no institution in the world that cannot
be improved with machine learning” .
Jeff Bezos
23. Machine Learning Platforms
An Infrastructure & Service layer to drive ML at scale in the enterprise
Facebook FBlearner May 9, 2016
https://code.fb.com/core-data/
introducing-fblearner-flow-facebook-s-
ai-backbone/
Google TFX Tensorflow Aug 13, 2017
https://www.tensorflow.org/tfx/
https://dl.acm.org/ft_gateway.cfm?
id=3098021&ftid=1899117&dwn=1&CF
ID=81485403&CFTOKEN=79729647b
2ac491f-EAC34BCC-93F2-A3C5-
BE9311C722468452
Netflix
Notebook Data
Platform
Aug 16, 2018 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Uber Michelangelo Sept 5, 2017 https://eng.uber.com/michelangelo/
Twitter Cortex Sept, 2015
https://cortex.twitter.com/en.html
https://blog.twitter.com/engineering/
en_us/topics/insights/2018/ml-
workflows.html
Magic Pony acquisition - 2016:
https://www.bernardmarr.com/
default.asp?contentID=1373
AirBnB BigHead Feb, 2018
https://databricks.com/session/
bighead-airbnbs-end-to-end-machine-
learning-platform
LinkedIN Pro-ML Oct, 2018
https://engineering.linkedin.com/blog/
2018/10/an-introduction-to-ai-at-
linkedin
25. Machine Learning Platforms
eBay Krylov Dec 17, 2019
https://tech.ebayinc.com/engineering/
ebays-transformation-to-a-modern-ai-
platform/
Lyft Flyte Jan 20, 2020
https://eng.lyft.com/introducing-flyte-
cloud-native-machine-learning-and-
data-processing-platform-
fb2bb3046a59
AT&T Acumos Oct 30, 2017 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Spotify
Spotify ML
platform
Dec 13, 2019
https://labs.spotify.com/2019/12/13/the-
winding-road-to-better-machine-
learning-infrastructure-through-
tensorflow-extended-and-kubeflow/
Delta Airlines (licensed) Jan 8, 2020
https://www.aviationtoday.com/
2020/01/08/delta-develops-ai-tool-
address-weather-disruption-improve-
flight-operations/
GE
Predix (customer
IoT platform)
Feb, 2018
https://www.ge.com/digital/sites/
default/files/download_assets/Predix-
The-Industrial-Internet-Platform-
Brief.pdf
KT Telecom Neuroflow Jan, 2018 https://disruptive.asia/kt-ai-platform-
internal-use/
An Infrastructure & Service layer to drive ML at scale in the enterprise
26. Machine Learning Platforms
Build vs Buy
The “custom build” approach, while highly customized to the needs of the organization, is
expensive, requires time and strong engineering talent and teams to develop and maintain it
The “buy” option often requires adapting to a given vendor’s approach but demands less time and
expertise and provides continued access to innovations
Ultimately, it’s a business case decision (ROI calculator next slide)
Partial list of ML platform licensees (courtesy of BigML Inc)
Most enterprises will ultimately implement
their ML platforms from commercial or
cloud-delivered software,
along with custom integration and custom-
coded modules tailored to their specific
needs
27. 28
ML Platform
Build vs Buy ROI
Source: Dataiku DS ROI toolkit https://pages.dataiku.com/data-science-roi-toolkit
28. 29
MACHINE LEARNING AS A SERVICE MACHINE LEARNING PLATFORM & SOFTWARE
https://www.crisp-research.com/vendor-universe/machine-learning/#fndtn-mlaas
Machine Learning Platforms
Vendor Landscape MLaaS: Machine Learning as a Service & On Premise
29. 30
ML Platformization Going Mainstream
Buy vs Build
Partial list of ML platform public customer references: HG Insights (BigML Inc, Dataiku & H2O.ai https://discovery.hgdata.com/product/bigml)
30. #MLSEV
ML platform Market References - a closer look
•Facebook - FBlearner
•Uber - Michelangelo
•AirBnB - BigHead
ML Platformization Going Mainstream: The Great Enterprise Pivot
31
Custom Built ML Platforms
Section 4
31. Facebook
FBlearner Flow: Facebook’s ML platform for internal use - May, 2016
25% of engineering team
using it
+1Mn ML models trained
+6 Mn predictions/sec
ML at scale:
Reusability
Parallelization
Simplicity
Automation
Rapid prototyping & experimentation
32. Facebook
FBlearner Flow: Facebook’s ML platform for internal use - May, 2016
Eliminating manual work for
experimentation
Engineers can spend more time
on feature engineering
which in turn produce greater
accuracy improvements
“
33. Uber
Michelangelo: Uber’s MLaaS platform for internal use - Sept, 2017
end-to-end ML workflow:
• manage data
• train
• evaluate
• deploy models
• make and monitor predictions.
Supports traditional ML models,
time series forecasting, and deep
learning.
35. AirBnB
Bighead - Feb, 2018
Airbnb’s internal ML platform is called Bighead.
Bighead is an end-to-end platform for building and deploying ML
models that aims to make the machine learning process at Airbnb
seamless, versatile, consistent, and scalable.
It is built in Python and relies on open source technology like
Docker, Jupyter, Spark, Kubernetes, and more.
These open source components are customized and integrated for
Airbnb’s specific needs. Like much of Airbnb’s technology
infrastructure, Bighead runs in AWS.
The platform was supported by an ML infrastructure team of 11
engineers and one product manager.
In the fall of 2018, Airbnb announced its plans to open source parts
of Bighead and Zipline in early 2019, but this hasn’t yet materialized.
36. The Great Pivot - ML at scale
Systems of Intelligence/ML drive efficiencies (1st), competitive advantages (2nd) & next
defensible business models ultimately
• Most large technology companies are
reconfiguring themselves around ML.
• Google was (arguably) the first company to
move, followed by Microsoft, Facebook,
Amazon, Apple and IBM.
• 2nd tier corporations following suit: GE, Uber,
even carriers as AT&T
• Not only a US phenomena - Alibaba, Baidu
chief Robin Li said in an internal memo that
Baidu’s strategic future relies on AI
• Ultimately all global players will need to re-tool
their processes adopting a ML driven
approach.h/t Jerry Chen - Greylock Partners
https://news.greylock.com/the-new-moats-53f61aeac2d9
37. #MLSEV
Scaling ML - Rapid Prototyping & AutoML:
Definition, Rationale
Vendor Comparison
AutoML - OptiML: Use Cases
38
Automated ML - AutoML
Section 6
39. AutoML
Automated Machine Learning
40
Problem Formulation
Data Acquisition
Feature Engineering
Modeling and Evaluations
Predictions
Measure Results
Data Transformations
5%
80%
• Data tasks, most consuming - Semi
automated.
• Feature Engineering is key to model
performance - semi automated
10% • Goal definition - Human driven
5%
• AutoML enables fast modeling/prototyping -
Automated
• Automated
40. 41
Enable knowledge workers (e.g., analysts, developers) to build stable and
insightful models quickly
Scale the number of predictive use cases in collaboration with non-technical
peers through quick prototyping.
Best AutoML approaches rely on automation of parts of the Machine Learning
process (e.g., hyper-parameter tuning) without limiting the practitioners’ ability
control customization.
GDPR, data privacy, interpretability and prediction explanations became
critical concerns when deploying AutoML
AutoML
Automated Machine Learning
42. 43
AutoML
Trade off in Model/Algorithm Selection
• Simple (Logistic Reg) vs
Complex (Deepnets, ANNs)
• Weak and Fast vs. Slow and
Robust
• Interpretability vs.
Representability
• Confidence vs. Performance
• Biased vs. Data-hungry
43. 44
AutoML DATAROBOT H2O BigML
Data Preparation
• Encoded categorical variables (one-hot);
Text n- grams; Missing values imputing;
Discretization (bins)
• limited manual transformations • Max. of
10 classes in the objective*
•Encoded categorical variables (one-hot); Missing
values handling; Date-time fields expansion; Bulk
interactions transformers; SVD numeric
transformer; CV target encoding; Cluster distance
transformer; Time lag
•Automatic feature engineering possible when
using AutoDL
• Encoded categorical variables (one-hot); Text
analysis; Missing values handling; Date-time fields
expansion
• Automatic Recursive Feature Selection & Feature
Engineering
• Multiple flexible manual transformations • Max of
1,000 classes in the objective
Optimization
Undisclosed optimization technique
(“expert data scientists preset
hyperparameter search space for models*)
Random Stacking
(a combination of random grid search and stacked
ensembles, plus early stopping)
Bayesian Parameter Optimization
(SMAC — Sequential Model-based Algorithm
Configuration) & DNN Metalearning
Models
•Open-source libraries: scikit-learn, R, H2O,
Tensorflow (not CNN or RNN), Spark,
XGBoost, DMTK, and Vowpal Wabbit
•They also “blend” multiple models during
the optimization process.
•GBMs, Random Forests, XGBoost, deep neural
nets, and extreme random forests
•· Stacks of models can be learned. Best of family
stacks adopt the top model type from each of the
main algorithms.
•Decision trees, random decision forests, boosting,
logistic regression, deep neural networks
•Customizable model ensembles with Fusions
leveraging the individually optimized models for
different classification, regression algorithms.
Speed It tests 30-40 different modeling
approaches and takes ~20 min.
Default time limit for AutoML is 1 hour. Can use
GPU or CPU. Can specify settings for accuracy,
time, and interpretability.
It tests 128 different modeling approaches
(creating more than 500 resources) and takes ~30
min.
Model
Visualizations &
Interpretability
• Limited model visualizations
• Feature importance for models • Predictions
explainability
• Dashboard: A single page with a global
interpretable model explanations plot, a feature
importance plot, a decision tree plot, and a partial
dependence plot.
• A machine learning interpretation tool (MLI) that
includes a KLIME or LIME-SUP graph.
• Multiple model visualizations to analyze the
impact of the variables on predictions:
sunburst, decision tree, partial dependence
plots, line chart (LR)
• Feature importance for models
• Predictions explainability
Model Evaluations
• Confusion matrix
• ROC curve (only for binary classification)
• Lift curve (only for binary classification)
• Side-by-side evaluations comparison
• Trade-off between complexity vs.
performance
• Models are ranked by cross-validation
AUC by default.
• Return leaderboard sortable by deviance (mean
residual deviance), logloss, MSE, RMSE, MAE,
RMSLE, mean per class error
• Confusion matrix
• ROC curve
• Precision-Recall curve
• Gain curve
• Lift curve
• Multiple evaluations comparison chart
Programmability &
Deployability
• Models can be used and created via API •
Export models
• Cloud, VPC or on-premises
• H2O allows you to convert the models you have
built to either a Plain Old Java Object (POJO) or a
Model ObJect, Optimized (MOJO).
• H2O-generated MOJO and POJO models are
ieasily embeddable in Java environments
• Models can be used and created via API • Export
models
• Cloud, VPC or on-premises
Source: Public Resources, Vendor Docs, BigML Analysis
Metalearning!
44. 45
AutoML - Metalearning
Automatic Network Hyperparameters Selection - DNNs (DeepNets)
We trained 296,748 deep neural networks
so you don’t have to!
• 296,748+ deep neural networks trained on 50 datasets
• For each one, recorded the optimum network structure for the
given dataset structure (number of fields, types of fields, etc)
• Trained a model to predict the optimum network structure for any
given dataset.
• This predicted network structure & hyper parameters can be
used directly or as a seed for a more intensive network search
Source: BigML - DeepNets https://blog.bigml.com/2017/10/04/deepnets-behind-the-scenes/
46. We are
here
(mostly)
Simplified* AI Technologies Landscape
* and imperfect
Future:
• Knowledge
representation
(symbolic/
Subsymbolic)
• Planning
(Reinforcement
Learning, Agents)
• Reasoning (Logic,
Symbolic)
• Search &
Optimization
(evolutionary/
genetic algos)
47. 48
BigML, IncPrivate and Confidential
BigML Product Progression
5
AutoML, Linear
Regression, Node-
Red, Workflow
Report, Improved
Topic Modeling
Organizations,
Operating
Thresholds, OptiML,
Fusions, Data
Transformations, PCA
Boosted Trees,
ROC Analysis,
Time Series,
DeepNets
Scripts, Libraries,
Executions,
WhizzML, Logistic
Regression, Topic
Models
Association
Discovery,
Correlations,
Samples,
Statistical Tests
Anomaly Detection,
Clusters, Flatline
Evaluations, Batch
Predictions,
Ensembles,
Starbursts
Core ML Workflow:
Source, Dataset,
Model, Prediction
Prototyping and
Beta
201920182017201620152014201320122011
Automating Model Creation, Selection, Operation and Workflows = Making Machine Learning Easier
Reproducibility at the core:
Programmability, Interpretability, Explainability are
essential part of BigML's platform
Sophistication
EaseofUse
WE HAVE BEEN BUILDING A STRONG FOUNDATION TO DEVELOP, DEPLOY AND OPERATE MACHINE-LEARNING BASED APPLICATIONS OF UNPARALLELED QUALITY
48. 49
BigML, IncPrivate and Confidential7
AI/MLMarketMaturity
Automating Workflows for
Model Creation,
Selection, Operation
Extending the Platform to Build and Manage Smarter Predictive Applications End-to-End
Building the BEST End-
to-End Machine
Learning Platform
2020 20301980
BigML's Co-Founder
Participates in first University
Machine Learning
2011
BigML
Founded
BigML Future
EXTENDING THE PLATFORM TO BUILD AND MANAGE SMARTER PREDICTIVE APPLICATIONS END-TO-END
Reasoning
Knowledge
Representation
Planning Optimization
Principles
Machine Learning
ROBUST AI
Doing to Reasoning, Planning, Knowledge Representation
and Optimization what we have done to Machine Learning
and combining them to build Robust AI Applications
Machine Learning
50. 51
1. ML platforms - Uber - Pooyan Jamshidi USC: https://pooyanjamshidi.github.io/mls/lectures/mls03.pdf
2. ML Systems - Jeff Smith (book)
3. Real World End to End ML: Srivatsan Srinivasan: https://www.slideshare.net/srivatsan88/real-world-end-to-end-machine-learning-pipeline-157130773
4. MLPaaS: https://thenewstack.io/an-introduction-to-the-machine-learning-platform-as-a-service/
5. NIPS: Hidden technical debt in ML: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
6. Twinml guide to AI platforms - Sam Charrington : https://twimlai.com/mlplatforms-ebook/
7. Carlos A. Gomez-Uribe and Neil Hunt, “The Netflix Recommender System: Algorithms, Business, Value, an Innovation,” ACM Transactions on Management Information Systems, January 2016, https://dl.acm.org/
citation.cfm?id=2843948.
8. Robert Chang, “Using Machine Learning to Predict Value of Home on Airbnb,” Medium, July 17, 2017, https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of- homes-on-
airbnb-9272d3d4739d.
9. Andrew Hoh and Nikhil Simha, “Zipline: Airbnb’s Machine Learning Data Management Platform,” SAIS 2018, June 12, 2018, https://databricks.com/session/zipline-airbnbs-machine-learning- data-management-
platform.
10.Jeffrey Dunn, “Introducing FBLearner Flow: Facebook’s AI Backbone,” Facebook Engineering, May 9, 2016, https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai- backbone.
11.Kim Hazelwood, et al, “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective,” Facebook, Inc., February 24, 2018, https://research.fb.com/wp-content/ uploads/2017/12/hpca-2018-
facebook.pdf.
12.Jermey Hermann and Mike Del Balso, “Meet Michelangelo: Uber’s Machine Learning Platform,” Uber Engineering, September 5, 2017, https://eng.uber.com/michelangelo/.
13.Kubeflow, “Kubeflow: Machine Learning Toolkit for Kubernetes,” Github, https://github.com/ kubeflow/.
14.James Kanter and Kalyan Veeramachaneni, “Deep Feature Synthesis: Towards Automating Data Science Endeavors,” 2015, http://www.jmaxkanter.com/static/papers/DSAA_DSM_2015.pdf.
15.Feature Labs, “Featuretools: An Open Source Python Framework for Automated Feature Engineering,” Github, https://github.com/featuretools/featuretools.
16.Frank Hutter, et al, “SMAC,” AutoML Freiburg-Hannover, https://www.automl.org/automated- algorithm-design/algorithm-configuration/smac/
17.Ruben Martinez-Cantin, “BayesOpt: A Toolbox for Bayesian Optimization, Experimental Design and Stochastic Bandits,” Github https://github.com/rmcantin/bayesopt.
18.Hyperopt, “Hyperopt: Distributed Asynchronous Hyperparameter Optimization in Python” Github, Septermber 4, 2011, https://github.com/hyperopt/hyperopt.
19.UCBerkeley RISELab, “Tune: Scalable Hyperparameter Search,” Github https://github.com/ ray-project/ray/tree/master/python/ray/tune.
20.Cade Metz, “Building A.I. That Can Build A.I.,” The New York Times, November 5, 2017, https:// www.nytimes.com/2017/11/05/technology/machine-learning-artificial-intelligence-ai.html
21.Google, “Cloud AutoML,” Google Cloud, https://cloud.google.com/automl/.
22.Neoklis Polyzotis, et al, “Data Management Challenges in Production Machine Learning,” Google, 2017, https://static.googleusercontent.com/media/research.google.com/en//pubs/ archive/46178.pdf.
23.Julien Simon, “Mastering the Mystical Art of Model Deployment,” Medium, July 28, 2018, https:// medium.com/faun/mastering-the-mystical-art-of-model-deployment-c0cafe011175.
24.Monica Rogati, “The AI Hierarchy of Needs,” Hackernoon, June 12, 2017, https://hackernoon. com/the-ai-hierarchy-of-needs-18f111fcc007.
List
of
References