Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/3aQXchV. Janet Bastiman describes how turning an AI proof of concept into a production ready, deployable system can be a world of pain, especially if different parts of the puzzle are fulfilled by different teams, going into technical details. Filmed at qconsf.com. Janet Bastiman is Chief Science Officer for Storystream where she heads up the AI strategy and also gets her hands dirty in the code with her team. She is also a Venture Partner at London based venture capital company MMC Ventures providing research and analysis on AI topics as well as advising portfolio companies on AI strategy.

1. 1StoryStream.ai
From POC to Production in
Minimal Time –
Avoiding Pain in ML Projects
Dr Janet Bastiman
@yssybyl

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
poc-ml/

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. 2StoryStream.ai
Project timings
Dr Janet Bastiman @yssybyl

5. 3StoryStream.ai
The world’s leading automotive content platform
StoryStream is a dedicated automotive content platform, trusted by some of the
world’s leading car brands. Specifically created to help automotive brands
provide a more relevant, engaging customer experience, fuelled with authentic
content and designed for efficiently scaling content operations across global
teams.
● Grow customer engagement and conversions by up to 25%
● Reduce content creation and management costs by up to 60%
● Provide a more authentic customer experience
● Understand your customer in a deeper way
About StoryStream
The Core StoryStream Benefits

6. 4StoryStream.ai

7. 5StoryStream.ai
Dr Janet Bastiman @yssybyl

8. 6StoryStream.ai
“[Client] needs this to go live at the end of
the month, I promised them we could
deliver...”
Every salesperson ever
Dr Janet Bastiman @yssybyl

9. 7StoryStream.ai
Project timings
Dr Janet Bastiman @yssybyl
● 35 models = 1050 days (one person linear)
● ~ 5 years for one person working Mon-Fri - who is allowed
holidays :)
● 250 days with parallelisation of tasks and data upfront
● 150 days on worksheet, balanced by an increase in ongoing
license

10. 8StoryStream.ai
Can you guess what happened next?
Dr Janet Bastiman @yssybyl

11. 9StoryStream.ai
What would it take to get it done in that time?
Dr Janet Bastiman @yssybyl
The Core (2003)
Paramount Pictures

12. 10StoryStream.ai
“They don’t have any data to give us”
Dr Janet Bastiman @yssybyl

13. 11StoryStream.ai
If you are dealing with any critical
inferencing do not take shortcuts, do it
properly and do it rigorously and stand up
to the company and say no - make sure
it’s clear that the timelines will be longer
to get it right.
Dr Janet Bastiman @yssybyl

14. 12StoryStream.ai
Without Data ML is just a Random Result
Dr Janet Bastiman @yssybyl
● Legal public sources
● https://github.com/awesomedata/awesome-public-datasets
● https://www.kaggle.com/datasets
● Take your own pictures/videos
● access/permission?
● Slow and inconsistent
● Scrape the client site with permission

15. 13StoryStream.ai
How much data?
Dr Janet Bastiman @yssybyl
• Vision: 1000 images per output class but depends on
complexity of the problem
• Time series: at least double the time period over which you
are predicting, but be cautious of data becoming irrelevant
• Text: very variable depending on the problem
• This also changes if you already have pre-trained networks
that you’re updating

16. 14StoryStream.ai
What do you do with the Data?
Dr Janet Bastiman @yssybyl
● Selection bias
● Random Sampling
● Over coverage
● Undercoverage
● Measurement (Response) error
● Processing errors
● Participation bias

17. 15StoryStream.ai
What do you do with the Data?
Dr Janet Bastiman @yssybyl
Photos
Scrape
S3 bucket ● Unique filename
● source
● Set uuid (if multiple images of
same car)
● Date taken
● S3 bucket per vehicle variant

18. 16StoryStream.ai
What do you do with the Data?
Dr Janet Bastiman @yssybyl
Photos
Scrape
Car
Detector
S3
Bucket
Manual
verification
● Extra field for label
● S3 bucket name became
mostly irrelevant

19. 17StoryStream.ai
Crowdsource labelling
Dr Janet Bastiman @yssybyl
https://xkcd.com/1897/

20. 19StoryStream.ai
Data Pipeline
Dr Janet Bastiman @yssybyl
Data In
Object
detector
Images
saved
Auxiliary
info saved
Temp public
access
Extract for
Turk
Import of
results
Dashboard
Expert
clean
Data
Ready

21. 21StoryStream.ai
Transfer Learning
Dr Janet Bastiman @yssybyl
● Use transfer learning - fix most of the weights of
a good network and adapt the last few layers
● Fast and easy retraining and works with smaller
data sets in a variety of fields
● (image) https://arxiv.org/abs/1903.02196
● (series) https://arxiv.org/abs/1907.01332
● (audio) https://arxiv.org/abs/1909.07526
Deep Learning for Vision Systems, Mohamed Elgendy

22. 22StoryStream.ai
Unbalanced Data
Dr Janet Bastiman @yssybyl

23. 23StoryStream.ai https://www.designhacks.co/products/cognitive-bias-codex-poster

24. 25StoryStream.ai
Stand on the shoulders
of giants…
Dr Janet Bastiman @yssybyl
● For some problems CNNs are robust to
noisy labels and up to 20 time noise to
real labels can still give business level
accuracy
https://arxiv.org/pdf/1705.10694.pdf
● Find the right architecture
http://www.asimovinstitute.org/neural-network-zoo/

25. 26StoryStream.ai
Go old school
Dr Janet Bastiman @yssybyl
Reduce the dimensionality of the problem and use Bayesian approach, KNN or SVM
https://xkcd.com/2059/

26. 27StoryStream.ai
Choose wisely
Dr Janet Bastiman @yssybyl

27. 28StoryStream.ai
Simplify the problem
Dr Janet Bastiman @yssybyl
Removal of camera artefacts in eye images to
make detection easier - Jeffrey De Fauw
http://blog.kaggle.com/2015/08/10/detecting-diabetic-
retinopathy-in-eye-images/
Image Image
Specific
Vehicle
Specific
Vehicle
Car?
Make?
Removal of Doppler effect on moving source using
fractional octave band shifting, F Mobley
https://asa.scitation.org/doi/pdf/10.1121/2.0000578?class=pdf
Δ𝑛=−r[𝑙𝑜𝑔2(1−𝑀cos𝜃sin𝜑)]

28. 29StoryStream.ai
Get every last drop from what you have
Dr Janet Bastiman @yssybyl
Statistical anatomical modelling for efficient and
personalised spine biomechanical models - I Castro
Mateos PhD thesis
Have a toolkit of augmentation
approaches but choose what’s relevant to
your needs...

29. 30StoryStream.ai
Augmentation - detail
Dr Janet Bastiman @yssybyl
● Flip L/R U/D
● Rotations
● Reduce or enlarge bounding box coordinates by N%
● Add occlusions
https://www.umbc.edu/rssipl/people/aplaza/Papers/Journals/2019
.GRSL.Occlusion.pdf
● Change hue saturation and value of colours in the image
https://arxiv.org/pdf/1902.06543.pdf
● Copypairing - https://arxiv.org/abs/1909.00390#

30. 34StoryStream.ai
Infrastructure
Dr Janet Bastiman @yssybyl
Data In Data Store
Taxonomy
Classifier
Definition
Test Set
DockerHub
Setup
Codeship
Project
GitHub
Setup
Notification
Slack
Email
Template
AWS
Image
Scripts
Dashboard

31. 35StoryStream.ai
Cloud Formation
Dr Janet Bastiman @yssybyl

32. 36StoryStream.ai
Automation
Dr Janet Bastiman @yssybyl
Delete
local data
Build
container
Get model
and key
Run test
harness
Validate
container
Run
container
Report
results
DashboardCommit
Build new
Container

33. 37StoryStream.ai
Stack Automation
Dr Janet Bastiman @yssybyl
Add new
container
Start stack
Run stack
test harness
Better?
Compare
results
Create docs
YesUpdate CFLive
No
Human
investigation

34. 38StoryStream.ai
Automatic Documentation
Dr Janet Bastiman @yssybyl
LaTeX
templates
Pweave
.tex files
and images
Save with
model files
Convert to
PDF
Run LaTeX
If live, save
in live docs
Email to
team

35. 40StoryStream.ai
Did we make it?
Dr Janet Bastiman @yssybyl
● Some really difficult images
● Only expected images were
given
● Where it was wrong it was
(mostly) sensibly wrong
● Client happy
● Cool automated system

36. 41StoryStream.ai
The Playbook
Dr Janet Bastiman @yssybyl
ai-playbook.com

37. 42StoryStream.ai
Dr Janet Bastiman @yssybyl
Thank You
https://xkcd.com/2191/

38. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
poc-ml/