1. The document describes Velox, a unified machine learning platform that aims to provide low latency predictions while models are continuously trained. 2. Velox uses a split model approach, where a shared basis feature model is trained in batch while personal user models are trained continuously online to provide personalized recommendations. 3. The system architecture includes a model manager that trains models, a prediction service that serves real-time queries from a frontend, and integrates with Spark for batch training.

1. VELOX: MODELS IN ACTION
Dan Crankshaw
UC Berkeley AMPLab
crankshaw@cs.berkeley.edu
Marin Software
2015

2. Algorithms, Machines, and People

3. Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

4. Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

5. Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

6. Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

7. GraphX
Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

8. GraphX
Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

9. GraphX
Algorithms, Machines, and People
“deep questions over dirty and heterogenous data”

10. BERKELEY DATA
ANALYTICS STACK (BDAS)
Spark
Spark
Streaming Spark SQL
BlinkDB
GraphX
MLlib
MLBase
HDFS, S3, …
Tachyon
Mesos HadoopYarn

11. Catify: Music for Cats

12. MODELINGTASK
Rating
Songs

13. MODELINGTASK
Ratings
Songs
Prediction

14. Catify: Music for Cats

15. Catify: Music for Cats
CatID Song Score
1 16 2.1
1 14 3.7
3 273 4.2
4 14 1.9

16. Catify: Music for Cats
Pipeline
CatID Song Score
1 16 2.1
1 14 3.7
3 273 4.2
4 14 1.9

17. Catify: Music for Cats
Tachyon + HDFS
Pipeline
CatID Song Score
1 16 2.1
1 14 3.7
3 273 4.2
4 14 1.9

18. Catify: Music for Cats
Tachyon + HDFS
Pipeline
CatID Song Score
1 16 2.1
1 14 3.7
3 273 4.2
4 14 1.9

19. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Catify: Music for Cats

20. Catify: Music for Cats
Songs
Users

21. Songs
Users
O(users * songs)
Catify: Music for Cats

22. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Catify: Music for Cats

23. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Precomputed
Ratings
Catify: Music for Cats

24. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Precomputed
Ratings
Catify: Music for Cats
Black box

25. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Training Data
Precomputed
Ratings
Catify: Music for Cats
Black box

26. Pipeline
Tachyon + HDFS
Node.js App Server
Apache Web Server
Training Data
Precomputed
Ratings
Catify: Music for Cats
Black box

27. What’s wrong?

28. 1. Serving system: low-latency but
high staleness
What’s wrong?

29. 1. Serving system: low-latency but
high staleness
2. Batch training: slow incremental
maintenance, no serving
What’s wrong?

30. 1. Serving system: low-latency but
high staleness
2. Batch training: slow incremental
maintenance, no serving
3. Ad-hoc model management
What’s wrong?

31. VELOX GOALS

32. VELOX GOALS
1. Low latency and fresh predictions

33. VELOX GOALS
1. Low latency and fresh predictions
2. Break the abstraction: model-
specific optimizations

34. VELOX GOALS
1. Low latency and fresh predictions
2. Break the abstraction: model-
specific optimizations
3. Unified system eases operation

35. Spark
Spark
Streaming Spark SQL
BlinkDB
GraphX
MLlib
MLBase
HDFS, S3, …
Tachyon
THE MISSING PIECE IN BDAS
Mesos

36. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Training
THE MISSING PIECE IN BDAS
Spark
HDFS, S3, …
Tachyon
Mesos

37. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Training Management + Serving
THE MISSING PIECE IN BDAS
Spark
HDFS, S3, …
Tachyon
Mesos

38. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Velox
Training Management + Serving
THE MISSING PIECE IN BDAS
Spark
HDFS, S3, …
Tachyon
Mesos

39. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Velox
Training Management + Serving
Spark
HDFS, S3, …
Tachyon
THE MISSING PIECE IN BDAS
Mesos

40. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Velox
Training Management + Serving
Spark
HDFS, S3, …
Tachyon
Model
Manager
THE MISSING PIECE IN BDAS
Mesos

41. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Velox
Training Management + Serving
Spark
HDFS, S3, …
Tachyon
Model
Manager
Prediction
Service
THE MISSING PIECE IN BDAS
Mesos

42. VELOX ARCHITECTURE

43. VELOX ARCHITECTURE
Standalone Scala
Service

44. VELOX ARCHITECTURE
Standalone Scala
Service
Automatic Integration
with Spark

45. VELOX ARCHITECTURE
Standalone Scala
Service
Personalized Predictions
as a Service
Automatic Integration
with Spark

46. VELOX ARCHITECTURE
Standalone Scala
Service
Shared-Nothing
Serving Cluster
Personalized Predictions
as a Service
Automatic Integration
with Spark

47. SYSTEM ARCHITECTURE
uuid: 01-10
uuid: 11-20
uuid: 20-30

48. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30

49. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

50. SYSTEM ARCHITECTURE
Predictions via REST
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

51. SYSTEM ARCHITECTURE
Predictions via REST
frontend.js
Returns score
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

52. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

53. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
Feedback via REST
uuid: 4

54. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
Feedback via REST
Model updated
in realtime
uuid: 4

55. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30

56. SYSTEM ARCHITECTURE
master
workerworker
worker
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30

57. SYSTEM ARCHITECTURE
master
workerworker
worker
Batch train RPC
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30

58. SYSTEM ARCHITECTURE
master
workerworker
worker
Batch train RPC
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
Returns batch
trained model

59. Mesos Mesos
HDFS, S3, …
Tachyon
HadoopYarn
Spark
Straming Shark
SQL
Graph
X ML
library
BlinkDB MLbase
Spark
Velox
Training Management + Serving
PREDICTION SERVICE
Model
Manager
Prediction
Service

60. PREDICTION API
GET
/velox/catify/predict?userid=22&song=27632
Simple point queries:

61. PREDICTION API
GET
/velox/catify/predict_top_k?userid=22&k=100
GET
/velox/catify/predict?userid=22&song=27632
Simple point queries:
More complex ordering queries:

62. PREDICTION API
GET
/velox/catify/predict_top_k?userid=22&k=100
GET
/velox/catify/predict?userid=22&song=27632
Simple point queries:
More complex ordering queries:
Low-latency and
scalable partitioning
Personalized
Predictions

63. PREDICTION API
GET
/velox/catify/predict_top_k?userid=22&k=100
GET
/velox/catify/predict?userid=22&song=27632
Simple point queries:
More complex ordering queries:
Low-latency and
scalable partitioning
Personalized
Predictions
Intelligent
Caching
Sharing and re-use of
model partial-state

64. SYSTEM ARCHITECTURE
Predictions via REST
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

65. PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
uuid model

66. PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
uuid model
Look up user
model
Read

67. PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
uuid model
Look up user
model
Primary key lookup
Read

68. PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
uuid model
Look up user
model
Primary key lookup
Partition queries by user:
always local
Read

69. Compute
Features
PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
user independent
}f( )

70. Compute
Features
PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
Feature computation
could be costly
user independent
}f( )

71. Compute
Features
PREDICTION EXECUTION
def
predict(
u:
UUID,
x:
Context
)
Feature computation
could be costly
user independent
}
Cache features for
reuse across users
f( )

72. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs

73. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords

74. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords
Candidate
Songs

75. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords
Candidate
Songs
Score and
rank all
candidates

76. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords
Candidate
Songs
By exploiting split model design we can leverage:
Score and
rank all
candidates

77. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords
Candidate
Songs
By exploiting split model design we can leverage:
Score and
rank all
candidates
A. Shrivastava, P. Li. “Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product
Search (MIPS).” NIPS’14 Best Paper

78. TOP-K QUERIES
Query predicate to pre-filter candidate set
All Songs Playlist Keywords
Candidate
Songs
By exploiting split model design we can leverage:
Score and
rank all
candidates
A. Shrivastava, P. Li. “Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product
Search (MIPS).” NIPS’14 Best Paper
Y. Low and A. X. Zheng. “Fast Top-K Similarity Queries Via Matrix Compression.” CIKM 2012

79. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

80. SYSTEM ARCHITECTURE
frontend.js
Returns score
uuid: 01-10
uuid: 11-20
uuid: 20-30
uuid: 4

81. Mesos Mesos
HDFS, S3, …
Tachyon
HadoopYarn
Spark
Straming Shark
SQL
Graph
X ML
library
BlinkDB MLbase
Spark
Velox
Training Management + Serving
Model
Manager
Prediction
Service
MODEL MANAGER

82. PERSONALIZED MODELING

83. PERSONALIZED MODELING

84. PERSONALIZED MODELING
A Separate Model for Each User?

85. PERSONALIZED MODELING
Computationally Inefficient
many complex models
A Separate Model for Each User?

86. PERSONALIZED MODELING
Statistically Inefficient
not enough data per user
Computationally Inefficient
many complex models
A Separate Model for Each User?

87. Input
(Song)
Rating

88. Input
(Song)
Rating

89. Input
(Song)
Rating
Split

90. Rating
Split
Input
(Song)

91. PERSONALIZED SPLIT MODEL
Input
(Song)

92. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model

93. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data

94. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly

95. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly
Train in Batch!

96. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly
Train in Batch!
Personalized
User Model

97. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly
Train in Batch!
Small Data
Personalized
User Model

98. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly
Train in Batch!
Small Data
Changes Quickly
Personalized
User Model

99. PERSONALIZED SPLIT MODEL
Input
(Song)
Shared Basis Feature Model
Big Data
Changes Slowly
Train in Batch!
Small Data
Changes Quickly
Train Online!
Personalized
User Model

100. Input
(Song)
Personalized
User Model
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

101. Input
(Song)
Personalized
User Model
Input
(Song)
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

102. Input
(Song)
Personalized
User Model
Input
(Song)
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

103. Input
(Song)
Personalized
User Model
Meow
Input
(Song)
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

104. Personalized
User Model
Meow
Input
(Song)
Input
(Song)
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

105. Personalized
User Model
Meow
Terrible
Input
(Song)
Input
(Song)
Shared Basis Feature Model
PERSONALIZED SPLIT MODEL

106. MATHEMATICAL FORMULATION
Input
(Song)

107. MATHEMATICAL FORMULATION
Input
(Song)
x

108. Shared Basis
Feature Models
Changes
slowly
MATHEMATICAL FORMULATION
Input
(Song)
x

109. Shared Basis
Feature Models
Changes
slowly
MATHEMATICAL FORMULATION
Input
(Song)
f(x; ✓)
x

110. Shared Basis
Feature Models
Personalized
User Model
Changes
slowly
Highly
dynamic
MATHEMATICAL FORMULATION
Input
(Song)
f(x; ✓)
x

111. Shared Basis
Feature Models
Personalized
User Model
Changes
slowly
Highly
dynamic
MATHEMATICAL FORMULATION
Input
(Song)
f(x; ✓) · wu
x

112. Shared Basis
Feature Models
Personalized
User Model
Changes
slowly
Highly
dynamic
= Rating
MATHEMATICAL FORMULATION
Input
(Song)
f(x; ✓) · wu
x
Meow

113. FEEDBACK API
POST
/velox/catify/observe?userid=22&song=27&score=3.7
Simple direct value feedback:

114. FEEDBACK API
POST
/velox/catify/observe?userid=22&song=27&score=3.7
Simple direct value feedback:
Continuously update
user models inVelox
Online Learning

115. FEEDBACK API
POST
/velox/catify/observe?userid=22&song=27&score=3.7
Simple direct value feedback:
Continuously update
user models inVelox
Online Learning Offline Learning
Logged toTachyon for
feature learning in Spark

116. FEEDBACK API
POST
/velox/catify/observe?userid=22&song=27&score=3.7
Simple direct value feedback:
Continuously update
user models inVelox
Online Learning Offline Learning
Logged toTachyon for
feature learning in Spark
Evaluation
Continuously assess
model performance

117. SYSTEM ARCHITECTURE
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30
Feedback via REST
Model updated
in realtime
uuid: 4

118. ONLINE LEARNING
velox.jar
user model
def
observe(u:
UUID,
x:
Context,
y:
Score)

119. ONLINE LEARNING
velox.jar
user model
def
observe(u:
UUID,
x:
Context,
y:
Score)
Update user
model with new
training data
Write

120. ONLINE LEARNING
velox.jar
user model
def
observe(u:
UUID,
x:
Context,
y:
Score)
Stochastic gradient descent
Update user
model with new
training data
Write

121. ONLINE LEARNING
velox.jar
user model
def
observe(u:
UUID,
x:
Context,
y:
Score)
Stochastic gradient descent
Incremental linear algebra
Update user
model with new
training data
Write

122. SYSTEM ARCHITECTURE
master
workerworker
worker
Batch train RPC
frontend.js
uuid: 01-10
uuid: 11-20
uuid: 20-30

123. OFFLINE OR NEARLINE
LEARNING
def
retrain(trainingData:
RDD)
Spark Based
Training Algs.wu · f(x; ✓)
Automated retraining policies
Efficient batch training using Spark
Incremental learning using Spark Streaming

124. Data Model

125. Data Model
Sample Bias: model affects the training data.

126. ALWAYS SERVETHE BEST SONG?
Songs
Predicted
Rating

127. ALWAYS SERVETHE BEST SONG?
Songs
Predicted
Rating

128. VELOX SOLUTION
Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song

129. VELOX SOLUTION
Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song

130. Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song
With prob. ϵ pick a random song
VELOX SOLUTION

131. Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song
With prob. ϵ pick a random song
VELOX SOLUTION

132. Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song
With prob. ϵ pick a random song
Epsilon Greedy
VELOX SOLUTION

133. Predicted
Rating
Songs
With prob. 1- ϵ serve the best predicted song
With prob. ϵ pick a random song
Epsilon Greedy
Active Learning
Opportunity to explore new systems for
this emerging analytics workload
VELOX SOLUTION

134. BEYOND RECOMMENDER
SYSTEMS

135. 1. Spam and anomaly detection
BEYOND RECOMMENDER
SYSTEMS

136. 1. Spam and anomaly detection
2. Device/location specific modeling
BEYOND RECOMMENDER
SYSTEMS

137. 1. Spam and anomaly detection
2. Device/location specific modeling
3. YOUR machine learning application
BEYOND RECOMMENDER
SYSTEMS

138. Spark
Streaming Spark
SQL
Graph
X ML
library
BlinkDB MLbase
Velox
Training Management + Serving
Spark
HDFS, S3, …
Tachyon
Model
Manager
Prediction
Service
THE MISSING PIECE IN BDAS
Mesos

139. SUMMARY

140. Today: model training and serving relies on ad-hoc,
manual processes spread across multiple systems
SUMMARY

141. Today: model training and serving relies on ad-hoc,
manual processes spread across multiple systems
TheVelox system automatically maintains multiple
models while providing low latency, fresh, and
personalized predictions
SUMMARY

142. Today: model training and serving relies on ad-hoc,
manual processes spread across multiple systems
TheVelox system automatically maintains multiple
models while providing low latency, fresh, and
personalized predictions
Velox will be open-source: coming soon to BDAS
SUMMARY

143. Today: model training and serving relies on ad-hoc,
manual processes spread across multiple systems
TheVelox system automatically maintains multiple
models while providing low latency, fresh, and
personalized predictions
Velox will be open-source: coming soon to BDAS
https://amplab.cs.berkeley.edu/projects/velox/
SUMMARY

144. Today: model training and serving relies on ad-hoc,
manual processes spread across multiple systems
TheVelox system automatically maintains multiple
models while providing low latency, fresh, and
personalized predictions
Velox will be open-source: coming soon to BDAS
https://amplab.cs.berkeley.edu/projects/velox/
crankshaw@cs.berkeley.edu
SUMMARY

145. QUESTIONS?