More Related Content Similar to AI & Machine Learning Web Day | Serverless Predictions at Scale Similar to AI & Machine Learning Web Day | Serverless Predictions at Scale (20) More from AWS Germany (20) AI & Machine Learning Web Day | Serverless Predictions at Scale1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Thomas Reske, AWS Solutions Architect
Serverless Predictions at Scale
08.05.2018
2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Erstellen und Ausführen
von Anwendungen
ohne sich über Server Gedanken
machen zu müssen
3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
$
Keine Serververwaltung Flexible Skalierung
Automatisierte
Hochverfügbarkeit
Keine ungenutzte Kapazität
Serverlose Anwendungen bieten wesentliche Vorteile
4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Prozesse rund um Maschinelles Lernen
Datenerhebung
Daten-
bereinigung und
-formatierung
Daten-
aufbereitung und
-transformation
Training
Evaluation
und
Verifikation
Integration
und
Deployment
Betrieb,
Monitoring
und Optimierung
5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Prozesse rund um Maschinelles Lernen
Datenerhebung
Daten-
bereinigung und
-formatierung
Daten-
aufbereitung und
-transformation
Training
Evaluation
und
Verifikation
Integration
und
Deployment
Betrieb,
Monitoring
und Optimierung
Wie kann ich effizient
vortrainierte Modelle für
Inferenzen nutzen?
Wie kann ich schnell
skalieren?
6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Was überführen wir eigentlich in
den Betrieb?
7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Ein Vortrainiertes Modell (Pre-trained Model)...
• ... ist ein Modell, das bereits trainiert wurde, z.T. sogar in einem anderen
Kontext (auf Basis anderer Daten)
• die Modell Artefakte (Struktur, Helper Files, Parameter/Gewichte, ...)
können mit den gängigen Deep Learning Frameworks sowohl serialisiert
und gespeichert werden als auch erneut deserialisiert und geladen werden
• dies erlaubt die Möglichkeit ein Fine-Tuning der Modelle durchzuführen
(„Head-Start“ oder „Freezing“), aber auch sie für ähnliche Probleme bzw.
andere Daten anzuwenden („Transfer Learning“)
• viele Modelle sind öffentlich verfügbar, z.B. im MXNet Model Zoo
8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Was fehlt?
9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Gibt es eine allgemeine Blaupause?
10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Model
Server
Model
Server
Modell
Server
Modell
Artefakte
Last-
verteilung
Generische Architektur
11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
1) Amazon API Gateway und Lambda Integration
Amazon API
Gateway
(API Proxy)
Amazon S3
(Modell Archiv)
AWS Lambda
(Datenverarbeitung)
12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Was sind die Trade-Offs?
13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Model Server for Apache MXNet
• Flexibles und einfach einzusetzendes Open-Source Projekt (Apache
License 2.0) um Anfragen an Deep Learning Modelle zu bedienen
• Unterstützt sowohl MXNet als auch Open Neural Network Exchange
(ONNX)
• Bietet CLI um Modell Artefakte zu paketieren und vorkonfigurierte Docker
Container um HTTP Endpunkte für Inferenzen zur Verfügung zu stellen
• Mehr Details unter https://github.com/awslabs/mxnet-model-server und
https://onnx.ai/
14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Quick Start - CLI
# create directory
mkdir -p ~/mxnet-model-server-quickstart && cd $_
# create and activate virtual environment
python3 -m venv ~/mxnet-model-server-quickstart/venv
source ~/mxnet-model-server-quickstart/venv/bin/activate
# install packages
pip install --upgrade pip
pip install mxnet-model-server
# run model server for Apache MXNet
mxnet-model-server --models
squeezenet=https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model
# classify an image
curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg
curl -s http://127.0.0.1:8080/squeezenet/predict -F "data=@kitten.jpg" | jq -r
15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Quick Start - Docker
# create directory
mkdir -p ~/mxnet-model-server-quickstart-docker && cd $_
# run model server for Apache MXNet
docker run -itd --name mms -p 8080:8080 awsdeeplearningteam/mms_cpu mxnet-model-server start --mms-
config /mxnet_model_server/mms_app_cpu.conf
# classify an image
curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg
curl -s http://127.0.0.1:8080/squeezenet/predict -F "data=@kitten.jpg" | jq -r
16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Warum Entwickler Container lieben…
Software
Paketierung
Verteilung “Immutable
Infrastructure”
17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Server
Guest OS
Bins/Libs Bins/Libs
App2App1
Einen einzelnen Container zu verwalten ist nicht schwer, …
18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Eine Containerflotte dagegen sehr…
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Verfügbarkeitszone 1 Verfügbarkeitszone 2
Verfügbarkeitszone 3
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Model
Server
Model
Server
Modell
Server
Modell
Artefakte
Last-
verteilung
Eine generische Architektur
20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2) AWS Fargate und Model Server für MXNet
AWS Fargate
(Datenverarbeitung
und Modell Archiv)
Elastic Load
Balancing
(Lastverteilung)
Amazon ECR
(Inferenz Code Image)
21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Was sind die Trade-Offs?
22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
3) Amazon SageMaker
Amazon S3
(Modell Archiv)
Amazon SageMaker
(ML Hosting Service)
Amazon ECR
(Inferenz Code Image)
23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Man muss sich aber nicht
auf eine Option festlegen!
24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon S3
(Modell Archiv)
AWS Fargate
(Datenverarbeitung)
Elastic Load
Balancing
(Lastverteilung)
Amazon API
Gateway
(API Proxy)
Amazon ECR
(Inferenz Code Image)
25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon S3
(Modell Archiv)
Amazon SageMaker
(ML Hosting Service)
Amazon ECR
(Inferenz Code Image)
Amazon API
Gateway
(API Proxy)
AWS Lambda
(Datenverarbeitung)
26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Zusammenfassung
Datenerhebung
Daten-
bereinigung und
-formatierung
Daten-
aufbereitung und
-transformation
Training
Evaluation
und
Verifikation
Integration
und
Deployment
Betrieb,
Monitoring
und Optimierung
Wie kann ich effizient
vortrainierte Modelle für
Inferenzen nutzen?
Wie kann ich schnell
skalieren?
27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Zusammenfassung
Datenerhebung
Daten-
bereinigung und
-formatierung
Daten-
aufbereitung und
-transformation
Training
Evaluation
und
Verifikation
Integration
und
Deployment
Betrieb,
Monitoring
und Optimierung
1
2
3
API Gateway + Lambda + S3
ELB + AWS Fargate
Amazon SageMaker
28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Weiterführende Ressourcen
• Serverlose Datenverarbeitung und serverlose Anwendungen
https://aws.amazon.com/de/serverless/
• Seamlessly Scale Predictions with AWS Lambda and MXNet
https://aws.amazon.com/blogs/compute/seamlessly-scale-predictions-with-aws-lambda-and-mxnet/
• Model Server for Apache MXNet
https://github.com/awslabs/mxnet-model-server
• Serverless Predictions at Scale
https://github.com/aws-samples/aws-ai-bootcamp-labs/blob/master/serverless_predictions.MD
• Serverless Inference with MMS on AWS Fargate
https://github.com/awslabs/mxnet-model-server/blob/master/docs/mms_on_fargate.md
• Introducing Model Server for Apache MXNet
https://aws.amazon.com/blogs/machine-learning/introducing-model-server-for-apache-mxnet/
• Using Chalice to serve SageMaker predictions
https://medium.com/@julsimon/using-chalice-to-serve-sagemaker-predictions-a2015c02b033
29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Go Build!
30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Thank you!