The document discusses using CI/CD techniques to industrialize machine learning workflows. It describes a typical ML workflow as being error-prone due to manual processes. The proposed solution is to use AWS services like SageMaker, CodeCommit, and S3 to create a flexible, collaborative, repeatable, auditable and secure ML workflow. A demo of the solution is shown and sample audit questions are provided that could be answered via AWS APIs.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Industrialize Machine Learning
Using CI/CD Techniques
Felix Candelario
Principal Global Accounts
Solutions Architect
AWS
F S V 3 0 4 - i
Harsha Sharma
Senior Technical Account Manager
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
The need for governing ML workflows
A typical ML workflow
Solution
Demo
Sample audit
Remarks

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highly regulated
workloads
Marketplace Lenders
• Operating exclusively online
• Niche product focus
• User of non-traditional data sources

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strong controls
required
Control objectives
• Govern the process while
maintaining flexibility
• Same inputs should result in same
outputs
• Significantly improve auditability
Risks
• Potentially greater exposure to bias
• Reproducibility is hard
• Field in great state of flux

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
On-Premise

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
Client
On-Premise
Idea
Network
drive
Source code
v13
GPU Cluster
Source code
v13

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
Client
On-Premise
Idea
Network
drive
GPU Cluster
Source code
v13

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
Client
On-Premise
Idea
Network
drive
GPU Cluster
Source code
v1v1

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
Client
On-Premise
Idea
Network
drive
GPU Cluster
Source code
v1

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical ML workflow
Client
On-Premise
Idea
Network
drive
Model weightGPU Cluster Training job
Source code
v1
Source code
v2
Training job
v1.1
Training job
Error prone
• Manual orchestration
• Inputs aren’t versions
• Dataset
• Source code
• Outputs aren’t versions
• Weights
• Gathering evidence is manual
• Who made the change?
• When was the change made?
• What version of data was used?

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution
Region
AWS Cloud

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution
Region
AWS Cloud
Source BucketBucket
Container
Training job
Input
Output
Dataset

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution
Region
AWS Cloud
BucketBucket
Container
Training job
Input
Output
Solution properties
• Flexible
• Collaborative
• Repeatable
• Auditable
• Secure

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile
• manifest.json
• train

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile
• manifest.json

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile
• manifest.json

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source
Input
Output
Dataset
Contents of repo
• buildspec.yaml
• Dockerfile
• manifest.json
• train

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
Source BucketBucket
Input
Output
Dataset

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
BucketBucket
Container
Input
Output

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
BucketBucket
Container
Training job
Input
Output

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
Region
AWS Cloud
BucketBucket
Container
Training job
Input
Output

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample audit
Region
AWS Cloud
Endpoints Inference
table
Apply for
loan
Trained model
Subsystem 1
Subsystem 2
Subsystem 3
Subsystem 4
Sample questions
• What version of the model produced
this inference?
• Who made the latest change to this
model?
• What version of training data was
exposed to this model?

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample audit
Region
AWS Cloud
Endpoints Inference
table
Apply for
loan
Trained model
Subsystem 1
Subsystem 2
Subsystem 3
Subsystem 4
Answers an API call away
sagemaker_client.describe_endpoint()
sagemaker_client.describe_endpoint_config
sagemaker_client.describe_model()
sagemaker_client.list_training_jobs()
codecommit_client.get_commit()

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Remarks
Region
AWS Cloud
BucketBucket
Container
Training job
Input
Output
Solution properties
• Flexible
• Collaborative
• Repeatable
• Auditable
• Secure

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Thursday, Nov 29
Build, Train, and Deploy ML Models Quickly and Easily with Amazon SageMaker, ft. Intuit
4:00 PM – 5:00 PM | Venetian, Level 3, San Polo 3405
Thursday, Nov 29
Build, Train, and Deploy ML Models with Amazon SageMaker
12:15 PM – 2:30 PM | Bellagio, Level 1, Grand Ballroom 9

47. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Felix Candelario
fcandela@amazon.com

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.