Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Model Serving for Deep Learning

1,692 views

Published on

Slides from my talk at the Data Innovations Summit on MXNet Model Server.
https://www.datainnovationsummit.com/

Apache MXNet Model Server (MMS) is a flexible and easy to use tool for serving deep learning models exported from MXNet or the Open Neural Network Exchange (ONNX).
https://github.com/awslabs/mxnet-model-server

Published in: Technology
  • Login to see the comments

Model Serving for Deep Learning

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Serving for Deep Learning ©2018 Amazon Web Services, Inc. or its affiliates, All rights reserved Adrian Hornsby, Technical Evangelist @adhorn
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are we talking about? AI Machine Learning Deep Learning
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is a Neural Net?
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Predicting the price of a house with humans Price City ZipCode Life Quality Parking Size # Room Accessibility Family Friendly
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Predicting the price of a house with neural network Price City ZipCode Life Quality Parking Size # Room Accessibility Family Friendly Input Output Discovered by the neural network
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning – Neural Networks Output Layer Input Layer Hidden Layers Many More…
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning is a Big Deal It’s able to do better than other ML and Humans
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://github.com/precedenceguo/mx-rcnn https://github.com/zhreshold/mxnet-yolo CNN: Object Detection
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://github.com/tornadomeet/mxnet-face CNN: Face Detection
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PredNet: Prediction Networks What comes next https://coxlab.github.io/prednet/
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CapsNet: Capsule Networks Spatial Memory https://arxiv.org/pdf/1710.09829v1.pdf
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Long Short Term Memory Networks (LSTM) https://github.com/awslabs/sockeye
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Generative Adversarial Networks (GAN) The future at work (already) today Generating new ”celebrity” faces https://github.com/tkarras/progressive_growing_of_gans
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Personalization Logistics Voice Autonomous Vehicles Deep Learning at Amazon
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How do people ”build” Neural Nets?
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Zoos & Transfer Learning • Full implementations of many state-of-the-art models reported in the academic literature. • Complete models, with scripts, pre-trained weights and instructions on how to build and fine tune these models.
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://www.youtube.com/watch?v=qGotULKg8e0 • Over 10 million images from 300,000 hotels • Fine-tuned a pre-trained Convolutional Neural Network using 100,000 images • Hotel descriptions now automatically feature the best available images Expedia Ranking hotel images using deep learning https://news.developer.nvidia.com/expedia-ranking-hotel-images-with-deep-learning/
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. So what does a deployed model looks like?
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Model Server Mobile Desktop IoT Internet
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Availability Networking Monitoring Model Decoupling Cross Framework Cross Platform The Undifferentiated Heavy Lifting of Model Serving
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tensor Flow Serving Model Server for MXNet UC Berkeley Clipper Model Serving Systems for Deep Learning
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Archive REST and OpenAPI Containerized ONNX Support Operational Metrics
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trained Network Model Signature Custom Code Auxiliary Assets Model Archive Model Export CLI Model Archive Back
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. REST and OpenAPI REST-like endpoint: <model-name>/predict Endpoint auto-generated from the model’s signature.json JSON encoding by default Binary input via request payload OpenAPI support – client code-gen and tooling Back
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Requests • Latencies • Resources Metrics • Model Name • Host Name Dimensions • Log / CSV • AWS CloudWatch Target Operational Metrics Back
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MMS Dockerfile Build Push Launch Containerization Container Cluster MMS Container MMS ContainerMMS Container MXNet NGINX MXNet Model Server Lightweight virtualization, isolation, runs anywhere Back
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. O(n2) Pairs MXNet Caffe2 PyTorch TF CNTKCoreML TensorRT NGraph SNPEMany Frameworks ONNX Support (initiative driven by AWS, Facebook and Microsoft) Many Platforms ONNX: Common IR Supported in MMS v0.2
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It’s Demo Time!
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open source – try it out and file issues github.com/awslabs/mxnet-model-server adhorn@amazon.com

×