Ellen Friedman and I spoke at the ACM meetup about how stream-first architecture can have a big impact and how the logistics of machine learning is a great example of that impact. This is my half of the presentation.

1. © 2017 MapR Technologies 1
Machine Learning Logistics

2. © 2017 MapR Technologies 2
Contact Information
Ted Dunning, PhD
Chief Application Architect, MapR Technologies
Committer, PMC member, board member, ASF
O’Reilly author
Email tdunning@mapr.com tdunning@apache.org
Twitter @Ted_Dunning

3. © 2017 MapR Technologies 3
Traditional View

4. © 2017 MapR Technologies 4
Traditional View: This isn’t the whole story

5. © 2017 MapR Technologies 5
90% of the effort in successful machine
learning isn’t in the training or model dev…
It’s the logistics

6. © 2017 MapR Technologies 6
Why?
• Just getting the training data is hard
– Which data? How to make it accessible? Multiple sources!
– New kinds of observations force restarts
– Requires a ton of domain knowledge
• The myth of the unitary model
– You can’t train just one
– You will have dozens of models, likely hundreds or more
– Handoff to new versions is tricky
– You have to get run-time to be sure about which is better


7. © 2017 MapR Technologies 7
What Machine Learning Tool is Best?
• Most successful groups keep several “favorite” machine
learning tools at hand
– No single tool is best in every situation
• The most important tool is a platform that supports logistics well
– Don’t have to do everything at the application level
– Lots of what matters can be handled at the platform level
• A good design for the logistics can make a big difference

8. © 2017 MapR Technologies 8
Some Gotchas
• Ops-oriented people will not “get it” regarding modeling
subtleties
• Data scientists will not “get it” regarding operational realities
• Therefore, modelers have to deliver self-contained models
• And, ops has to provide pre-wired structure

9. © 2017 MapR Technologies 9
Rendezvous Architecture
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results

10. © 2017 MapR Technologies 10
Rendezvous to the Rescue: Better ML Logistics
• Stream-1st architecture is a powerful approach with surprisingly
widespread advantages
– Innovative technologies emerging to for streaming data
• Microservices approach provides flexibility
– Streaming supports microservices (if done right)
• Containers remove surprises
– Predictable environment for running models

11. © 2017 MapR Technologies 11
Rendezvous: Mainly for Decisioning Engines
• Decisioning models
– Looking for a “right answer”
– Simpler than reinforcement learning
• Examples include:
– Fraud detection
– Predictive analytics / market prediction
– Churn prediction (as in telecommunications)
– Yield optimization
– Deep learning in form of speech or image recognition, in some cases

12. © 2017 MapR Technologies 12
What We Ultimately Want
request
response
Model

13. © 2017 MapR Technologies 13
But This Isn’t The Answer
Model 1
request
response
Load
balancer
Model 2
Model 3

14. © 2017 MapR Technologies 14
First Try with Streams
Input
Model 1
Model 2
Model 3
request
response
?

15. © 2017 MapR Technologies 15
First Rendezvous
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results

16. © 2017 MapR Technologies 16
Some Key Points
• Note that all models see identical inputs
• All models run in production setting
• All models send scores to same stream
• The rendezvous server decides which scores to ignore
• Roll forward, roll back, correlated comparison are all now trivial

17. © 2017 MapR Technologies 17
Reality Check, Injecting External State
Model 1
Model 2
Model 3
request
Raw
Add
external
data
Input
Database
The world

18. © 2017 MapR Technologies 18
Recording Raw Data (as it really was)
Input
Scores
Decoy
Model 2
Model 3
Archive

19. © 2017 MapR Technologies 19
Quality & Reproducibility of Input Data is Important!
• Recording raw-ish data is really a big deal
– Data as seen by a model is worth gold
– Data reconstructed later often has time-machine leaks
– Databases were made for updates, streams are safer
• Raw data is useful for non-ML cases as well (think flexibility)
• Decoy model records training data as seen by models under
development & evaluation

20. © 2017 MapR Technologies 20
Canary for Comparison
Real
model
∆
Result
Canary
Decoy
Archive
Input

21. © 2017 MapR Technologies 21
What Does the Canary Do?
• The canary is a real model, but is very rarely updated
• The canary results are almost never used for decisioning
• The virtue of the canary is stability
• Comparing to the canary results gives insight into new models

22. © 2017 MapR Technologies 22
Isolated Development With Stream Replication
Model 1
Model 2
Model 3
request
Raw
Add
external
data
Input
Internal 1
Internal 2
Internal 3
The world
Model 4
Raw
New
external
data
Input
Internal 4
Production
Development

23. © 2017 MapR Technologies 23
A Quick Review
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

24. © 2017 MapR Technologies 24
The Proxy Talks to the Outside World
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

25. © 2017 MapR Technologies 25
The Input Stream Feeds All Models Identically
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

26. © 2017 MapR Technologies 26
The Scores Stream Contains All Results
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

27. © 2017 MapR Technologies 27
The Rendezvous Picks A Result
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

28. © 2017 MapR Technologies 28
Results Return Via A Stream and Return Address
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

29. © 2017 MapR Technologies 29
Models in production live in the real
world:
Conditions may (will) change

30. © 2017 MapR Technologies 30
Rendezvous Schedules
• The key idea of rendezvous schedules is to define the trade-off
of latency versus model priority
– At short delays, we want the best
– At moderate delays we will compromise a bit
– Near the deadline, we will take any answer at all
• Normally the same rendezvous schedules apply to all
transactions
– Overriding default schedule has bona fide uses

31. © 2017 MapR Technologies 31
Rendezvous Overrides
• Incoming transaction can carry an overriding schedule
– This is great for QA, to see output from a specific model
– Overriding the default schedule is also good for systemic A/B tests
• Overrides should be unusual

32. © 2017 MapR Technologies 32
Scaling Up
• More kinds of model
– multiple rendezvous frameworks for different tasks
• More throughput
– Fast default models
– Partition input stream to allow parallel model evaluation
– Input batching
• Extreme volumes require extreme measures
– Cannibalize fancy models to run more fast/simple models
– Speed before beauty

33. © 2017 MapR Technologies 33
Faster Throughput Through Failure
• Suppose we have one model that can handle 10,000 t/s @ 2ms
– But this isn’t the most accurate model. Not bad, but not best
• And our champion model can handle 1000 t/s @ 10ms
• Then imagine a burst of 2000 t/s for several minutes
• Champion can only evaluate half of all requests
– Should skip to keep up
– Fast model will cover for champion

34. © 2017 MapR Technologies 34
Input Scores
Model 1
Model 2
Model 3

35. © 2017 MapR Technologies 35
Input Scores
Model 1
Model 2
Model 3

36. © 2017 MapR Technologies 36
Input Scores
Model 1
Model 2
Model 3

37. © 2017 MapR Technologies 37
Always have a default or
fallback model
Models that fall behind should
discard requests to catch up

38. © 2017 MapR Technologies 38
Limitations of Rendezvous
• 100% speculative execution can be expensive
– Can be mitigated by partial speculation
– Or it may just be too expensive
• Minimum Viable Products should be minimal
– You may not require zero downtime … be realistic
• Context may be too large
• Latency limits may be too stringent

39. © 2017 MapR Technologies 39
Ad Targeting Example
Detailed
scoring
Proxy Pre-select
1
2
Sharded Ad Scoring
3
User
Profile
Ads
User profile and context used
for rough-cut selection of ads
Roughly 1000 ads are scored in
detail for p(click)

40. © 2017 MapR Technologies 40
Why Not Full Rendezvous?
• 1000’s of ads / second x 1000 candidates = 1M scores /
second
– AKA “a lot”
• Scoring a single model is expensive
• Sharding and replication provides a form of failure tolerance
• Full speculative execution across several options is prohibitive
• Latency guarantees can be very short (10 ms)

41. © 2017 MapR Technologies 41
Rendezvous-lite Options
• We have some options
• We can allow selective speculation on marked requests
– If only 1% of ads run speculative execution, we can pack 10x more
shards per node and use 10x fewer nodes
– Selective speculation doesn’t give redundancy
• We can release results if >80% of shards reply
• Temporary speculation during hand-offs is useful

42. © 2017 MapR Technologies 42
Let’s Review

43. © 2017 MapR Technologies 43
A Quick Review
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

44. © 2017 MapR Technologies 44
The Proxy Talks to the Outside World
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

45. © 2017 MapR Technologies 45
The Input Stream Feeds All Models Identically
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

46. © 2017 MapR Technologies 46
The Scores Stream Contains All Results
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

47. © 2017 MapR Technologies 47
The Rendezvous Picks A Result
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

48. © 2017 MapR Technologies 48
Results Return Via A Stream and Return Address
Input Scores
RendezvousModel 1
Model 2
Model 3
request
response
Results
Proxy

49. © 2017 MapR Technologies 49
Not Such Bad Ideas
• Keep models running “in the wings”
– Don’t wait until conditions change to start building the next model
– Keep new short-history models ready to roll, some graybeards as well
• Hot hand-off
– With rendezvous: just stop ignoring the new best model
• Deploy a canary server
– Keep an old model active as a reference
– If it was 90% correct, difference with any better model should be small
– Score distribution should be roughly constant

50. © 2017 MapR Technologies 50
New book: how to manage machine learning models
Download free pdf or read free online via @MapR:
https://mapr.com/ebook/machine-learning-logistics/
“Rendezvous Architecture” by Ted Dunning & Ellen Friedman, in
Encyclopedia of Big Data Technologies. Sherif Sakr and Albert
Zomaya, editors. Springer International Publishing, in press 2018.
and

51. © 2017 MapR Technologies 51
Contact Information
Ted Dunning, PhD
Chief Application Architect, MapR Technologies
Committer, PMC member, board member, ASF
O’Reilly author
Email tdunning@mapr.com tdunning@apache.org
Twitter @Ted_Dunning

52. © 2017 MapR Technologies 52
Q&A
@mapr
tdunning@mapr.com
ENGAGE WITH US
@ Ted_Dunning