Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2017 MapR Technologies
Applying Machine Learning to IOT:
End to End Distributed Pipeline for Real-
Time Uber Data Using ...
© 2017 MapR Technologies
Uber trip cluster dashboard
© 2017 MapR Technologies
Data Collect Process Store
Spark
Streami
ng
Analyze
HBase
SQL
ML
Model
Stream
Input
Spark
Streami...
© 2017 MapR Technologies
Fast data Pipeline for Uber Data Using Apache APIs: Kafka, Spark, Hba
•  Why combine Machine Lear...
© 2017 MapR Technologies
Why combine Streaming events
with Machine Learning?
© 2017 MapR Technologies
What’s a Stream ?
Producers ConsumersEvents_Stream
an unbounded sequence of events
Events
© 2017 MapR Technologies
Why Stream Processing?
6:05 P.M.: 90°
To
pic
Stream
Temperature
Turn on the
air
conditioning!
It’...
© 2017 MapR Technologies
Why combine Streaming Events with Machine Learning?
Fraud detection Smart Machinery Utility Smart...
© 2017 MapR Technologies
Why combine IOT/Streaming Events with Machine Learning?
•  Audi and Daimler deep learning for
aut...
© 2017 MapR Technologies
Why combine IOT with Machine Learning?
•  Monitoring devices combined with ML can provide alerts ...
© 2017 MapR Technologies
Why combine IOT with Machine Learning?
•  A Stanford team has shown that a machine-learning model...
© 2017 MapR Technologies
Applying Machine Learning to Live Patient Data
•  https://www.slideshare.net/caroljmcdonald/apply...
© 2017 MapR Technologies
What if BP had detected problems before the oil hit the water ?
•  1M samples/sec
•  High perform...
© 2017 MapR Technologies
Why combine IOT/Streaming Events with Machine Learning?
•  ML & location data:
–  identify behavi...
© 2017 MapR Technologies
Why combine IOT with Machine Learning?
•  Uber Near Realtime Price Surging
–  https://www.slidesh...
© 2017 MapR Technologies
What has changed in the past 10 years?
•  Distributed computing
•  Streaming analytics
•  Improve...
© 2017 MapR Technologies
Intro to Machine Learning
© 2017 MapR Technologies
What is Machine Learning?
Data Build ModelTrain Algorithm
Finds patterns
New Data Use Model
(pred...
© 2017 MapR Technologies
ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set...
© 2017 MapR Technologies
End to End Application Architecture
© 2017 MapR Technologies
What is Supervised Machine Learning?
Supervised
•  Classification
–  Naïve Bayes
–  SVM
–  Random...
© 2017 MapR Technologies
Supervised Algorithms use labeled data
Data
features
Build Model
New Data
features
Predict
Use Mo...
© 2017 MapR Technologies
Supervised Machine Learning: Classification & Regression
Classification
Identifies
category for i...
© 2017 MapR Technologies
Classification: Definition
Form of ML that:
•  Identifies which category an item belongs to
•  Us...
© 2017 MapR Technologies
If it Walks/Swims/Quacks Like a Duck …… Then It Must Be a Duck
swims
walks
quacks
Features:
walks...
© 2017 MapR Technologies
Debit Card Fraud Example
•  What are we trying to predict?
–  This is the Label or Target outcome...
© 2017 MapR Technologies
Decision Tree For Classification
•  Tree of decisions about features
•  IF THEN ELSE questions us...
© 2017 MapR Technologies
What is Unsupervised Machine Learning?
Machine Learning
Unsupervised
•  Clustering
–  K-means
•  ...
© 2017 MapR Technologies
Unsupervised Algorithms use Unlabeled data
Customer GroupsBuild ModelTrain Algorithm
Finds patter...
© 2017 MapR Technologies
Unsupervised Machine Learning: Clustering
Clustering
group news articles into different categories
© 2017 MapR Technologies
Clustering: Definition
•  Unsupervised learning task
•  Groups objects into clusters of high simi...
© 2017 MapR Technologies
Clustering: Definition
•  Unsupervised learning task
•  Groups objects into clusters of high simi...
© 2017 MapR Technologies
Clustering: Example
•  Group similar objects
© 2017 MapR Technologies
Clustering: Example
•  Group similar objects
•  Use MLlib K-means algorithm
1.  Initialize coordi...
© 2017 MapR Technologies
Clustering: Example
•  Group similar objects
•  Use MLlib K-means algorithm
1.  Initialize coordi...
© 2017 MapR Technologies
Clustering: Example
•  Group similar objects
•  Use MLlib K-means algorithm
1.  Initialize coordi...
© 2017 MapR Technologies
Clustering: Example
•  Group similar objects
•  Use MLlib K-means algorithm
1.  Initialize coordi...
© 2017 MapR Technologies
Spark intro
© 2017 MapR Technologies
Apache Spark Streaming
•  Task scheduling
•  Memory Management
•  Fault recovery
•  Interacting w...
© 2017 MapR Technologies
Spark Distributed Datasets
Dataset
W
Executor
P4
W
Executor
P1 P3
W
Executor
P2
partitioned
Parti...
© 2017 MapR Technologies
Example loading a Dataset
val df: Dataset[Uber] = spark.read.option("inferSchema",
"false").schem...
© 2017 MapR Technologies
Example:
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
tasks
tasks
tasks
© 2017 MapR Technologies
Example
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
Read
HDFS
Block
Read
HDFS
Block
Read
...
© 2017 MapR Technologies
Example
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
Cache 1
Cache 2
Cache 3
Process
& Cac...
© 2017 MapR Technologies
Example:
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
Cache 1
Cache 2
Cache 3
results
resu...
© 2017 MapR Technologies
Example
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
Cache 1
Cache 2
Cache 3
val df: Datas...
© 2017 MapR Technologies
Example
Worker
Worker
Worker
Block 1
Block 2
Block 3
Cache 1
Cache 2
Cache 3
tasks
tasks
tasks
Dr...
© 2017 MapR Technologies
Example: Log Mining
Worker
Worker
Worker
Block 1
Block 2
Block 3
Cache 1
Cache 2
Cache 3
Driver
P...
© 2017 MapR Technologies
Example:
Worker
Worker
Worker
Block 1
Block 2
Block 3
Cache 1
Cache 2
Cache 3
Driver
results
resu...
© 2017 MapR Technologies
Example:
Worker
Worker
Worker
Block 1
Block 2
Block 3
Cache 1
Cache 2
Cache 3
Driver
Cache your d...
© 2017 MapR Technologies
Spark Use Cases
Iterative Algorithms on large amounts of data
Some Algorithms that need iteration...
© 2017 MapR Technologies
Cluster Uber Trip Locations
© 2017 MapR Technologies
Part 1: Spark Machine Learning
•  End to End Application for Monitoring Uber Data using Spark ML
...
© 2017 MapR Technologies
Zeppelin Notebook with Spark
Data
Engineer
Data
Scientist
© 2017 MapR Technologies
Spark ML workflow
© 2017 MapR Technologies
Uber Data
•  Date/Time: The date and time of the Uber pickup
•  Lat: The latitude of the Uber pic...
© 2017 MapR Technologies
Load the data into a Dataframe: Define the Schema
case class Uber(dt: String, lat: Double, lon: D...
© 2017 MapR Technologies
Dataset merged with Dataframe
•  in Spark 2.0, DataFrame APIs merged with Datasets APIs
•  A Data...
© 2017 MapR Technologies
Data
Frame
Load data
Load the data into a Dataframe
val schema = StructType(Array(
StructField("d...
© 2017 MapR Technologies
Load the data into a Dataframe
Dataframe
row
columns
© 2017 MapR Technologies
Dataset
Load data
Load the data into a Dataset
case class Uber(dt: String, lat: Double, lon: Doub...
© 2017 MapR Technologies
Load the data into a Dataset
Dataset
Collection of Uber
objects
columns
row
© 2017 MapR Technologies
Uber Example
•  What are the “if questions” or
properties we can use to group?
–  These are the F...
© 2017 MapR Technologies
Extract the Features
Image reference O’Reilly Learning Spark
+
+
̶+
̶ ̶
Feature Vectors Model
Fea...
© 2017 MapR Technologies
Use VectorAssembler to put features in vector column
val featureCols = Array("lat", "lon")
val as...
© 2017 MapR Technologies
Data
Frame
Load data transform
Estimator
val kmeans = new KMeans()
.setK(8)
.setFeaturesCol("feat...
© 2017 MapR Technologies
Data
Frame
Load data transform
Estimator
val model = kmeans.fit(df2)
Fit the Model on the Trainin...
© 2017 MapR Technologies
Data
Frame
Load data transform
Estimator
model.clusterCenters.foreach(println)
[40.76930621976264...
© 2017 MapR Technologies
fitted
model
Analyze Clusters
summary
val clusters = model.summary.predictions
clusters.show()
pr...
© 2017 MapR Technologies
fitted
model
Transform new data, adds column with Clusters
transform
features
val clusters = mode...
© 2017 MapR Technologies
fitted
model
Save the model to distributed file system
save
model.write.overwrite().save("/path/s...
© 2017 MapR Technologies
Kafka API and Streaming Data
© 2017 MapR Technologies
Part 2: MapR Event Streams with Kafka API and Spark Streaming
•  End to End Application for Monit...
© 2017 MapR Technologies
Serve DataStore DataCollect Data
What Do We Need to Do ?
Process DataData Sources
? ? ? ?
© 2017 MapR Technologies
Collect the Data
Data IngestSource
Stream
Topic
•  Data Ingest:
–  Network Based: MapR Event
Stre...
© 2017 MapR Technologies
Organize Data into Topics with MapR Streams
Topics Organize Events into Categories and Decouple P...
© 2017 MapR Technologies
Scalable Messaging with MapR Streams
Server 1
Partition1: Topic - Pressure
Partition1: Topic - Te...
© 2017 MapR Technologies
Scalable Messaging with MapR Streams
Partition1: Topic - Pressure
Partition1: Topic - Temperature...
© 2017 MapR Technologies
Scalable Messaging with MapR Streams
Partition1: Topic - Pressure
Partition1: Topic - Temperature...
© 2017 MapR Technologies
Partition is like a Queue
Consumers
MapR Cluster
Topic: Admission / Server 1
Topic: Admission / S...
© 2017 MapR Technologies
Events are delivered in the order they are received, like a queue
messages are delivered in the o...
© 2017 MapR Technologies
Unlike a queue, events are persisted even after they’re delivered
Messages remain on the partitio...
© 2017 MapR Technologies
How do we do this with High Performance at Scale?
•  Parallel operations
•  minimizes disk read/w...
© 2017 MapR Technologies
Processing Same Message for Different Purposes
Consumers
Consumers
Consumers
Producers
Producers
...
© 2017 MapR Technologies
Machine Learning Logistics
Input Data +
Actual Delay
Input Data +
Predictions
Consumer
withML
Mod...
© 2017 MapR Technologies
Use the Model with Streaming Data
© 2017 MapR Technologies
Collect Data
Process the Data with Spark Streaming and Spark Machine Learning
Process Data
Stream...
© 2017 MapR Technologies
ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set...
© 2017 MapR Technologies
Use Case: Real-Time Analysis of Geographically Clustered Vehicles
Uber trip data enrich with K-me...
© 2017 MapR Technologies
Use Case: Time Series Data
Uber trip data
Stream
Topic
2014-08-01 00:00:00,
40.729,-73.9422,B0259...
© 2017 MapR Technologies
Processing Spark DStreams
Data stream divided into batches of X milliseconds = DStreams
© 2017 MapR Technologies
Load the saved model
// load model for getting clusters
val model = KMeansModel.load(modelpath)
© 2017 MapR Technologies
Create a DStream
DStream: a sequence of RDDs
representing a stream of data
val messagesDStream = ...
© 2017 MapR Technologies
Parse message txt to Uber Object and convert to DataFrame
uDStream.foreachRDD{ rdd =>
// get clus...
© 2017 MapR Technologies
Enrich Data with Cluster
© 2017 MapR Technologies
Convert to JSON send to Topic, Send the Enriched Message
© 2017 MapR Technologies
Process Dstream Streaming Applicaton Output
dStream
batch
time 2 to 3
batch
time 1 to 2
batch
tim...
© 2017 MapR Technologies
Real Time Dashboard
© 2017 MapR Technologies
Part 3: Realtime Dashboard using Vert.x
•  End to End Application for Monitoring Uber Data using ...
© 2017 MapR Technologies
Serve DataCollect Data
Serving the Data
MapR-FS
Process DataData Sources
Stream
Topic
© 2017 MapR Technologies
Use Case: Real-Time Analysis of Geographically Clustered Vehicles
Uber trip data enrich with K-me...
© 2017 MapR Technologies
Use Case Dashboard
© 2017 MapR Technologies
The Vert.x toolkit and Web Application Architecture
•  Event-driven
•  Event Bus
•  Verticles sin...
© 2017 MapR Technologies
Dashboard Architecture
© 2017 MapR Technologies
The Dashboard Vert.x HTML5 Javascript Client
© 2017 MapR Technologies
Initializing the Heatmap
© 2017 MapR Technologies
Dashboard Architecture
© 2017 MapR Technologies
Creating the Vertx EventBus
•  create an instance of the vertx.EventBus object
•  add an onopen l...
© 2017 MapR Technologies
Add Event Trip location points to Map
Parse JSON message
© 2017 MapR Technologies
Add Event Trip location points to Map
Add lattitude and longitude points to heatmap
© 2017 MapR Technologies
Add Event Trip location points to Map
If cluster center is new then add marker
© 2017 MapR Technologies
Spark and HBase
© 2017 MapR Technologies
Part 4: using MapR-DB with HBase API
•  https://mapr.com/blog/monitoring-uber-pt4/
© 2017 MapR Technologies
Serve DataStore DataCollect Data
What Do We Need to Do ?
MapR-FS
Process DataData Sources
MapR-FS...
© 2017 MapR Technologies
Use Case: Real-Time Analysis of Geographically Clustered Vehicles
Uber trip data enrich with K-me...
© 2017 MapR Technologies
MapR-DB (HBase API) is Designed to Scale
Key
Range
xxxx
xxxx
Key
Range
xxxx
xxxx
Key
Range
xxxx
x...
© 2017 MapR Technologies
Store Lots of Data with NoSQL MapR-DB
bottleneck
Storage ModelRDBMS MapR-DB
Normalized schema à J...
© 2017 MapR Technologies
HBase Schema
With Hbase/MapR-DB data is automatically partitioned by Key Range
© 2017 MapR Technologies
Spark Streaming writing to MapR-DB (HBase API)
© 2017 MapR Technologies
Spark HBase and MapR-DB Binary Connector
•  HConnection object in every Spark Executor:
•  allowi...
© 2017 MapR Technologies
Spark Hbase streamBulkPut
•  HBaseContext streamBulkPut method parameters:
•  message value DStre...
© 2017 MapR Technologies
Massively Parrallel writes to HBase
The Spark Streaming bulk put enables massively parallel sendi...
© 2017 MapR Technologies
HBase Schema
To use the Spark HBase Connector for reads, you need to define the Catalog for the
s...
© 2017 MapR Technologies
SparkSQL and DataFrames: Define the Schema
define the Catalog for the schema mapping between the ...
© 2017 MapR Technologies
Loading data from MapR-DB into a Spark DataFrame
Use Catalog defining schema
© 2017 MapR Technologies
Spark Dataframes combine filters and select
filters rows for cluster ids (the beginning of the ro...
© 2017 MapR Technologies
Use Case: Real-Time Data Pipelines
Input Data +
Actual Delay
Input Data +
Predictions
Consumer
wi...
© 2017 MapR Technologies
© 2017 MapR Technologies
To Learn More:
•  MapR Free ODT http://learn.mapr.com/
© 2017 MapR Technologies
MapR Blog
• https://www.mapr.com/blog/
© 2017 MapR Technologies
MapR Container for Developers
• https://maprdocs.mapr.com/home/MapRContainerDevelopers/
MapRConta...
© 2017 MapR Technologies
…helping you put data technology to work
●  Find answers
●  Ask technical questions
●  Join on-de...
© 2017 MapR Technologies
Stream Processing
Building a Complete Data Architecture
MapR File System
(MapR-XD)
MapR Converged...
© 2017 MapR Technologies
Q&A
ENGAGE WITH US
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

5

Share

Download to read offline

Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using Apache APIs: Kafka, Spark, HBase

Download to read offline

Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using Apache APIs: Kafka, Spark, HBase

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using Apache APIs: Kafka, Spark, HBase

  1. 1. © 2017 MapR Technologies Applying Machine Learning to IOT: End to End Distributed Pipeline for Real- Time Uber Data Using Apache APIs: Kafka, Spark, HBase Carol McDonald @caroljmcdonald
  2. 2. © 2017 MapR Technologies Uber trip cluster dashboard
  3. 3. © 2017 MapR Technologies Data Collect Process Store Spark Streami ng Analyze HBase SQL ML Model Stream Input Spark Streami ng Stream Enriched Use Case: Real-Time Analysis of Geographically Clustered Vehicles
  4. 4. © 2017 MapR Technologies Fast data Pipeline for Uber Data Using Apache APIs: Kafka, Spark, Hba •  Why combine Machine Learning with Streaming Events? •  Introduction to Machine Learning and Spark •  Introduction to Kafka & Spark Streaming •  ISpark Streaming and NoSQL HBase Note: this code example is from me, only the data is from Uber
  5. 5. © 2017 MapR Technologies Why combine Streaming events with Machine Learning?
  6. 6. © 2017 MapR Technologies What’s a Stream ? Producers ConsumersEvents_Stream an unbounded sequence of events Events
  7. 7. © 2017 MapR Technologies Why Stream Processing? 6:05 P.M.: 90° To pic Stream Temperature Turn on the air conditioning! It’s becoming important to process events as they arrive
  8. 8. © 2017 MapR Technologies Why combine Streaming Events with Machine Learning? Fraud detection Smart Machinery Utility Smart Meters Home Automation Networks Manufacturing Security Systems Patient Monitoring
  9. 9. © 2017 MapR Technologies Why combine IOT/Streaming Events with Machine Learning? •  Audi and Daimler deep learning for autonomous vehicles –  Using MapR platform to scale deep learning efforts https://mapr.com/company/press-releases/norcom-selects-mapr-deep- learning/
  10. 10. © 2017 MapR Technologies Why combine IOT with Machine Learning? •  Monitoring devices combined with ML can provide alerts for Sepsis, which is one of the leading causes for death in hospitals –  http://www.computerweekly.com/news/450422258/Putting-sepsis-algorithms-into-electronic- patient-records
  11. 11. © 2017 MapR Technologies Why combine IOT with Machine Learning? •  A Stanford team has shown that a machine-learning model can identify heart arrhythmias from an electrocardiogram (ECG) better than an expert –  https://www.technologyreview.com/s/608234/the-machines-are-getting-ready-to-play-doctor/
  12. 12. © 2017 MapR Technologies Applying Machine Learning to Live Patient Data •  https://www.slideshare.net/caroljmcdonald/applying-machine-learning-to- live-patient-data
  13. 13. © 2017 MapR Technologies What if BP had detected problems before the oil hit the water ? •  1M samples/sec •  High performance at scale is necessary!
  14. 14. © 2017 MapR Technologies Why combine IOT/Streaming Events with Machine Learning? •  ML & location data: –  identify behavior patterns and trends: –  telecom, travel, marketing... •  http://www.cisco.com/c/en/us/solutions/ industries/smart-connected-communities.html
  15. 15. © 2017 MapR Technologies Why combine IOT with Machine Learning? •  Uber Near Realtime Price Surging –  https://www.slideshare.net/ConfluentInc/kafka-uber- the-worlds-realtime-transit-infrastructure-aaron- schildkrout •  (however this example code is mine) NEAR REALTIME PRICE SURGING
  16. 16. © 2017 MapR Technologies What has changed in the past 10 years? •  Distributed computing •  Streaming analytics •  Improved machine learning
  17. 17. © 2017 MapR Technologies Intro to Machine Learning
  18. 18. © 2017 MapR Technologies What is Machine Learning? Data Build ModelTrain Algorithm Finds patterns New Data Use Model (prediction function) Predictions Contains patterns Recognizes patterns
  19. 19. © 2017 MapR Technologies ML Discovery Model Building Model Training/ Building Training Set Test Model Predictions Test Set Evaluate Results Historical Data Deployed Model Insights Data Discovery, Model Creation Production Feature Extraction Feature Extraction ●  Churn Modelling Uber trips Stream TopicUber trips New Data
  20. 20. © 2017 MapR Technologies End to End Application Architecture
  21. 21. © 2017 MapR Technologies What is Supervised Machine Learning? Supervised •  Classification –  Naïve Bayes –  SVM –  Random Decision Forests •  Regression –  Linear –  Logistic Machine Learning Unsupervised •  Clustering –  K-means •  Dimensionality reduction –  Principal Component Analysis –  SVD
  22. 22. © 2017 MapR Technologies Supervised Algorithms use labeled data Data features Build Model New Data features Predict Use Model X1, X2 Y f(X1, X2) =Y X1, X2 Y
  23. 23. © 2017 MapR Technologies Supervised Machine Learning: Classification & Regression Classification Identifies category for item
  24. 24. © 2017 MapR Technologies Classification: Definition Form of ML that: •  Identifies which category an item belongs to •  Uses supervised learning algorithms –  Data is labeled Sentiment
  25. 25. © 2017 MapR Technologies If it Walks/Swims/Quacks Like a Duck …… Then It Must Be a Duck swims walks quacks Features: walks quacks swims Features:
  26. 26. © 2017 MapR Technologies Debit Card Fraud Example •  What are we trying to predict? –  This is the Label or Target outcome: –  Fraud or Not Fraud •  What are the “if questions” or properties we can use to predict? –  These are the Features: –  Is the amount spent today > historical average? –  Unusual region for card history ? –  Known merchant or not ?
  27. 27. © 2017 MapR Technologies Decision Tree For Classification •  Tree of decisions about features •  IF THEN ELSE questions using features at each tree node •  Answers branch to child nodes Is the amount spent in 24 hours > average Is the number of states used from > 2 Are there multiple Purchases today from risky merchants? YES NO NoYES Fraud 90% Not Fraud 50% Fraud 90% Not Fraud 30% YES No
  28. 28. © 2017 MapR Technologies What is Unsupervised Machine Learning? Machine Learning Unsupervised •  Clustering –  K-means •  Dimensionality reduction –  Principal Component Analysis –  SVD Supervised •  Classification –  Naïve Bayes –  SVM –  Random Decision Forests •  Regression –  Linear –  Logistic
  29. 29. © 2017 MapR Technologies Unsupervised Algorithms use Unlabeled data Customer GroupsBuild ModelTrain Algorithm Finds patterns New Customer Purchase Data Use Model Similar Customer Group Contains patterns Recognizes patterns Customer purchase data
  30. 30. © 2017 MapR Technologies Unsupervised Machine Learning: Clustering Clustering group news articles into different categories
  31. 31. © 2017 MapR Technologies Clustering: Definition •  Unsupervised learning task •  Groups objects into clusters of high similarity
  32. 32. © 2017 MapR Technologies Clustering: Definition •  Unsupervised learning task •  Groups objects into clusters of high similarity –  Search results grouping –  Grouping of customers, patients –  Text categorization –  recommendations •  Anomaly detection: find what’s not similar
  33. 33. © 2017 MapR Technologies Clustering: Example •  Group similar objects
  34. 34. © 2017 MapR Technologies Clustering: Example •  Group similar objects •  Use MLlib K-means algorithm 1.  Initialize coordinates to center of clusters (centroid) x x x x x
  35. 35. © 2017 MapR Technologies Clustering: Example •  Group similar objects •  Use MLlib K-means algorithm 1.  Initialize coordinates to center of clusters (centroid) 2.  Assign all points to nearest centroid x x x x x
  36. 36. © 2017 MapR Technologies Clustering: Example •  Group similar objects •  Use MLlib K-means algorithm 1.  Initialize coordinates to center of clusters (centroid) 2.  Assign all points to nearest centroid 3.  Update centroids to center of points x x x x x
  37. 37. © 2017 MapR Technologies Clustering: Example •  Group similar objects •  Use MLlib K-means algorithm 1.  Initialize coordinates to center of clusters (centroid) 2.  Assign all points to nearest centroid 3.  Update centroids to center of points 4.  Repeat until conditions met x x x x x
  38. 38. © 2017 MapR Technologies Spark intro
  39. 39. © 2017 MapR Technologies Apache Spark Streaming •  Task scheduling •  Memory Management •  Fault recovery •  Interacting with storage systems •  DataFrame API •  Catalyst Optimizer •  Processing of live streams •  Micro-batching •  Machine Learning •  Multiple types of ML algorithms •  Graph processing •  Graph parallel computations Distributed Parallel Cluster computing Programming Framework
  40. 40. © 2017 MapR Technologies Spark Distributed Datasets Dataset W Executor P4 W Executor P1 P3 W Executor P2 partitioned Partition 1 8213034705, 95, 2.927373, jake7870, 0…… Partition 2 8213034705, 115, 2.943484, Davidbresler2, 1…. Partition 3 8213034705, 100, 2.951285, gladimacowgirl, 58… Partition 4 8213034705, 117, 2.998947, daysrus, 95…. •  Read only collection of typed objects Dataset[T] •  Partitioned across a cluster •  Operated on in parallel •  in memory can be Cached
  41. 41. © 2017 MapR Technologies Example loading a Dataset val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count Worker Worker Worker Driver Block 1 Block 2 Block 3
  42. 42. © 2017 MapR Technologies Example: Worker Worker Worker Block 1 Block 2 Block 3 Driver tasks tasks tasks
  43. 43. © 2017 MapR Technologies Example Worker Worker Worker Block 1 Block 2 Block 3 Driver Read HDFS Block Read HDFS Block Read HDFS Block
  44. 44. © 2017 MapR Technologies Example Worker Worker Worker Block 1 Block 2 Block 3 Driver Cache 1 Cache 2 Cache 3 Process & Cache Data Process & Cache Data Process & Cache Data
  45. 45. © 2017 MapR Technologies Example: Worker Worker Worker Block 1 Block 2 Block 3 Driver Cache 1 Cache 2 Cache 3 results results results val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count res9: Long = 829275
  46. 46. © 2017 MapR Technologies Example Worker Worker Worker Block 1 Block 2 Block 3 Driver Cache 1 Cache 2 Cache 3 val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count df.show
  47. 47. © 2017 MapR Technologies Example Worker Worker Worker Block 1 Block 2 Block 3 Cache 1 Cache 2 Cache 3 tasks tasks tasks Driver
  48. 48. © 2017 MapR Technologies Example: Log Mining Worker Worker Worker Block 1 Block 2 Block 3 Cache 1 Cache 2 Cache 3 Driver Process from Cache Process from Cache Process from Cache Cached, does not have to read from file again val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count df.show
  49. 49. © 2017 MapR Technologies Example: Worker Worker Worker Block 1 Block 2 Block 3 Cache 1 Cache 2 Cache 3 Driver results results results val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count df.show
  50. 50. © 2017 MapR Technologies Example: Worker Worker Worker Block 1 Block 2 Block 3 Cache 1 Cache 2 Cache 3 Driver Cache your data è Faster Results val df: Dataset[Uber] = spark.read.option("inferSchema", "false").schema(schema).csv(“data/uber.csv").as[Uber] df.cache df.count df.show
  51. 51. © 2017 MapR Technologies Spark Use Cases Iterative Algorithms on large amounts of data Some Algorithms that need iterations •  Clustering (K-Means) •  Linear Regression •  Graph Algorithms (e.g., PageRank) •  Alternating Least Squares ALS Some Example Use Cases: •  Anomaly detection •  Classification •  Recommendations
  52. 52. © 2017 MapR Technologies Cluster Uber Trip Locations
  53. 53. © 2017 MapR Technologies Part 1: Spark Machine Learning •  End to End Application for Monitoring Uber Data using Spark ML •  https://mapr.com/blog/monitoring-real-time-uber-data-using-spark-machine- learning-streaming-and-kafka-api-part-1/
  54. 54. © 2017 MapR Technologies Zeppelin Notebook with Spark Data Engineer Data Scientist
  55. 55. © 2017 MapR Technologies Spark ML workflow
  56. 56. © 2017 MapR Technologies Uber Data •  Date/Time: The date and time of the Uber pickup •  Lat: The latitude of the Uber pickup •  Lon: The longitude of the Uber pickup •  Base: The TLC base company affiliated with the Uber pickup The Data Records are in CSV format. An example line is shown below: •  2014-08-01 00:00:00,40.729,-73.9422,B02598
  57. 57. © 2017 MapR Technologies Load the data into a Dataframe: Define the Schema case class Uber(dt: String, lat: Double, lon: Double, base: String) val schema = StructType(Array( StructField("dt", TimestampType, true), StructField("lat", DoubleType, true), StructField("lon", DoubleType, true), StructField("base", StringType, true) )) Input Comma Separated Values: datetime, lattitude, longitude, base 2014-08-01 00:00:00,40.729,-73.9422,B02598
  58. 58. © 2017 MapR Technologies Dataset merged with Dataframe •  in Spark 2.0, DataFrame APIs merged with Datasets APIs •  A Dataset is a collection of typed objects (SQL and functions) •  A DataFrame is a Dataset of generic Row objects (SQL)
  59. 59. © 2017 MapR Technologies Data Frame Load data Load the data into a Dataframe val schema = StructType(Array( StructField("dt", TimestampType, true), StructField("lat", DoubleType, true), StructField("lon", DoubleType, true), StructField("base", StringType, true) )) val df = spark.read.option("inferSchema", "false").schema(schema) .csv("/user/user01/data/uber.csv") df.show
  60. 60. © 2017 MapR Technologies Load the data into a Dataframe Dataframe row columns
  61. 61. © 2017 MapR Technologies Dataset Load data Load the data into a Dataset case class Uber(dt: String, lat: Double, lon: Double, base: String) extends Serializable val schema = StructType(Array( StructField("dt", TimestampType, true), StructField("lat", DoubleType, true), StructField("lon", DoubleType, true), StructField("base", StringType, true) )) val df = spark.read.option("inferSchema", "false").schema(schema) .csv("/user/user01/data/uber.csv") .as[Uber] df.show
  62. 62. © 2017 MapR Technologies Load the data into a Dataset Dataset Collection of Uber objects columns row
  63. 63. © 2017 MapR Technologies Uber Example •  What are the “if questions” or properties we can use to group? –  These are the Features: –  We will group by Lattitude, longitude •  Use Spark SQL to analyze: Day of the week, time, rush hour … NEAR REALTIME PRICE SURGING
  64. 64. © 2017 MapR Technologies Extract the Features Image reference O’Reilly Learning Spark + + ̶+ ̶ ̶ Feature Vectors Model Featurization Training Model Evaluation Best Model Training Data + + ̶+ ̶ ̶ + + ̶+ ̶ ̶ + + ̶+ ̶ ̶ + + ̶+ ̶ ̶ Feature Vectors are vectors of numbers representing the value for each feature
  65. 65. © 2017 MapR Technologies Use VectorAssembler to put features in vector column val featureCols = Array("lat", "lon") val assembler = new VectorAssembler() .setInputCols(featureCols) .setOutputCol("features") val df2 = assembler.transform(df) Data Frame Load data transform DataFrame + Features
  66. 66. © 2017 MapR Technologies Data Frame Load data transform Estimator val kmeans = new KMeans() .setK(8) .setFeaturesCol("features") .setMaxIter(5) Create Kmeans Estimator, Set Features DataFrame + Features
  67. 67. © 2017 MapR Technologies Data Frame Load data transform Estimator val model = kmeans.fit(df2) Fit the Model on the Training Data Features DataFrame + Features fit fitted model input
  68. 68. © 2017 MapR Technologies Data Frame Load data transform Estimator model.clusterCenters.foreach(println) [40.76930621976264,-73.96034885367698] [40.67562793272868,-73.79810579052476] [40.68848772848041,-73.9634449047477] [40.78957777777776,-73.14270740740741] [40.32418330308531,-74.18665245009073] [40.732808848486286,-74.00150153727878] [40.75396549974632,-73.57692359208531] [40.901700842900674,-73.868760398198] Clusters from fitted model DataFrame + Features fit fitted model input
  69. 69. © 2017 MapR Technologies fitted model Analyze Clusters summary val clusters = model.summary.predictions clusters.show() prediction DataFrame + Features + prediciton
  70. 70. © 2017 MapR Technologies fitted model Transform new data, adds column with Clusters transform features val clusters = model.transform(newdata) prediction DataFrame + Features DataFrame + Features + prediciton
  71. 71. © 2017 MapR Technologies fitted model Save the model to distributed file system save model.write.overwrite().save("/path/savemodel") Use later val sameModel = KMeansModel.load("/user/user01/data/savemodel") DataFrame + Features
  72. 72. © 2017 MapR Technologies Kafka API and Streaming Data
  73. 73. © 2017 MapR Technologies Part 2: MapR Event Streams with Kafka API and Spark Streaming •  End to End Application for Monitoring Uber Data using Spark ML •  https://mapr.com/blog/monitoring-real-time-uber-data-using-spark-machine- learning-streaming-and-kafka-api-part-2/
  74. 74. © 2017 MapR Technologies Serve DataStore DataCollect Data What Do We Need to Do ? Process DataData Sources ? ? ? ?
  75. 75. © 2017 MapR Technologies Collect the Data Data IngestSource Stream Topic •  Data Ingest: –  Network Based: MapR Event Streams using the Kafka API
  76. 76. © 2017 MapR Technologies Organize Data into Topics with MapR Streams Topics Organize Events into Categories and Decouple Producers from Consumers Consumers MapR Cluster Topic: Pressure Topic: Temperature Topic: Warnings Consumers Consumers Kafka API Kafka API
  77. 77. © 2017 MapR Technologies Scalable Messaging with MapR Streams Server 1 Partition1: Topic - Pressure Partition1: Topic - Temperature Partition1: Topic - Warning Server 2 Partition2: Topic - Pressure Partition2: Topic - Temperature Partition2: Topic - Warning Server 3 Partition3: Topic - Pressure Partition3: Topic - Temperature Partition3: Topic - Warning Topics are partitioned for throughput and scalability
  78. 78. © 2017 MapR Technologies Scalable Messaging with MapR Streams Partition1: Topic - Pressure Partition1: Topic - Temperature Partition1: Topic - Warning Partition2: Topic - Pressure Partition2: Topic - Temperature Partition2: Topic - Warning Partition3: Topic - Pressure Partition3: Topic - Temperature Partition3: Topic - Warning Producers are load balanced between partitions Kafka API
  79. 79. © 2017 MapR Technologies Scalable Messaging with MapR Streams Partition1: Topic - Pressure Partition1: Topic - Temperature Partition1: Topic - Warning Partition2: Topic - Pressure Partition2: Topic - Temperature Partition2: Topic - Warning Partition3: Topic - Pressure Partition3: Topic - Temperature Partition3: Topic - Warning Consumers Consumers Consumers Consumer groups can read in parallel Kafka API
  80. 80. © 2017 MapR Technologies Partition is like a Queue Consumers MapR Cluster Topic: Admission / Server 1 Topic: Admission / Server 2 Topic: Admission / Server 3 Consumers Consumers Partition 1 New Messages are appended to the end Partition 2 Partition 3 6 5 4 3 2 1 3 2 1 5 4 3 2 1 Producers Producers Producers New Message 6 5 4 3 2 1 Old Message
  81. 81. © 2017 MapR Technologies Events are delivered in the order they are received, like a queue messages are delivered in the order they are received MapR Cluster 6 5 4 3 2 1 Consumer groupProducers Read cursors Consumer group
  82. 82. © 2017 MapR Technologies Unlike a queue, events are persisted even after they’re delivered Messages remain on the partition, available to other consumers MapR Cluster (1 Server) Topic: Warning Partition 1 3 2 1 Unread Events Get Unread 3 2 1 Client Library ConsumerPoll
  83. 83. © 2017 MapR Technologies How do we do this with High Performance at Scale? •  Parallel operations •  minimizes disk read/writes
  84. 84. © 2017 MapR Technologies Processing Same Message for Different Purposes Consumers Consumers Consumers Producers Producers Producers MapR-FS Kafka API Kafka API
  85. 85. © 2017 MapR Technologies Machine Learning Logistics Input Data + Actual Delay Input Data + Predictions Consumer withML Model 2 Consumer withML Model 1 Decoy results Consumer Consumer withML Model 3 Consumer Stream Archive Stream Scores Stream Input SQL SQL Real time Data Stream Input Delayed data Input Data + Predictions + Actual Delay Real Time dashboard + Historical Analysis
  86. 86. © 2017 MapR Technologies Use the Model with Streaming Data
  87. 87. © 2017 MapR Technologies Collect Data Process the Data with Spark Streaming and Spark Machine Learning Process Data Stream Topic •  Extension of the core Spark AP •  scalable, high-throughput, fault- tolerant stream processing
  88. 88. © 2017 MapR Technologies ML Discovery Model Building Model Training/ Building Training Set Test Model Predictions Test Set Evaluate Results Historical Data Deployed Model Insights Data Discovery, Model Creation Production Feature Extraction Feature Extraction Uber trips Stream TopicUber trips New Data
  89. 89. © 2017 MapR Technologies Use Case: Real-Time Analysis of Geographically Clustered Vehicles Uber trip data enrich with K-means Cluster location Stream Topic Stream Topic Spark Streaming Spark Streaming Write to MapR-DB SQL
  90. 90. © 2017 MapR Technologies Use Case: Time Series Data Uber trip data Stream Topic 2014-08-01 00:00:00, 40.729,-73.9422,B02598 {"dt":"2014-08-01 00:00:00.0”, "lat":40.3495,"lon":-74.0667, "base":"B02682","cluster":5} Enrich with K-means cluster id Spark Streaming read Stream Topic
  91. 91. © 2017 MapR Technologies Processing Spark DStreams Data stream divided into batches of X milliseconds = DStreams
  92. 92. © 2017 MapR Technologies Load the saved model // load model for getting clusters val model = KMeansModel.load(modelpath)
  93. 93. © 2017 MapR Technologies Create a DStream DStream: a sequence of RDDs representing a stream of data val messagesDStream = KafkaUtils.createDirectStream[String,String] (ssc, LocationStrategies.PreferConsistent,consumerStrategy) // get message values from key,value and parse to Uber objects val uDStream = linesDStream.map(_.value()) batch time 0 to 1 batch time 1 to 2 batch time 2 to 3 dStream Stored in memory as an RDD
  94. 94. © 2017 MapR Technologies Parse message txt to Uber Object and convert to DataFrame uDStream.foreachRDD{ rdd => // get cluster centers and add to df // send to Topic } ssc.start() ssc.awaitTermination()
  95. 95. © 2017 MapR Technologies Enrich Data with Cluster
  96. 96. © 2017 MapR Technologies Convert to JSON send to Topic, Send the Enriched Message
  97. 97. © 2017 MapR Technologies Process Dstream Streaming Applicaton Output dStream batch time 2 to 3 batch time 1 to 2 batch time 0 to 1 Result Dstream Transformed RDDs map map map Stream Topic
  98. 98. © 2017 MapR Technologies Real Time Dashboard
  99. 99. © 2017 MapR Technologies Part 3: Realtime Dashboard using Vert.x •  End to End Application for Monitoring Uber Data using Spark ML •  https://mapr.com/blog/monitoring-uber-with-spark-streaming-kafka-and- vertx/
  100. 100. © 2017 MapR Technologies Serve DataCollect Data Serving the Data MapR-FS Process DataData Sources Stream Topic
  101. 101. © 2017 MapR Technologies Use Case: Real-Time Analysis of Geographically Clustered Vehicles Uber trip data enrich with K-means Cluster location Stream Topic Stream Topic Spark Streaming Spark Streaming Write to MapR-DB SQL
  102. 102. © 2017 MapR Technologies Use Case Dashboard
  103. 103. © 2017 MapR Technologies The Vert.x toolkit and Web Application Architecture •  Event-driven •  Event Bus •  Verticles single threaded
  104. 104. © 2017 MapR Technologies Dashboard Architecture
  105. 105. © 2017 MapR Technologies The Dashboard Vert.x HTML5 Javascript Client
  106. 106. © 2017 MapR Technologies Initializing the Heatmap
  107. 107. © 2017 MapR Technologies Dashboard Architecture
  108. 108. © 2017 MapR Technologies Creating the Vertx EventBus •  create an instance of the vertx.EventBus object •  add an onopen listener, which registers an event bus handler for the address “dashboard.” •  handler will receive all messages published to the “dashboard” address
  109. 109. © 2017 MapR Technologies Add Event Trip location points to Map Parse JSON message
  110. 110. © 2017 MapR Technologies Add Event Trip location points to Map Add lattitude and longitude points to heatmap
  111. 111. © 2017 MapR Technologies Add Event Trip location points to Map If cluster center is new then add marker
  112. 112. © 2017 MapR Technologies Spark and HBase
  113. 113. © 2017 MapR Technologies Part 4: using MapR-DB with HBase API •  https://mapr.com/blog/monitoring-uber-pt4/
  114. 114. © 2017 MapR Technologies Serve DataStore DataCollect Data What Do We Need to Do ? MapR-FS Process DataData Sources MapR-FS Stream Topic
  115. 115. © 2017 MapR Technologies Use Case: Real-Time Analysis of Geographically Clustered Vehicles Uber trip data enrich with K-means Cluster location Stream Topic Stream Topic Spark Streaming Spark Streaming Write to MapR-DB SQL
  116. 116. © 2017 MapR Technologies MapR-DB (HBase API) is Designed to Scale Key Range xxxx xxxx Key Range xxxx xxxx Key Range xxxx xxxx Fast Reads and Writes by Key! Data is automatically partitioned by Key Range! Key colB colC xxx val val xxx val val Key colB colC xxx val val xxx val val Key colB colC xxx val val xxx val val
  117. 117. © 2017 MapR Technologies Store Lots of Data with NoSQL MapR-DB bottleneck Storage ModelRDBMS MapR-DB Normalized schema à Joins for queries can cause bottleneck De-Normalized schema à Data that is read together is stored together Key colB colC xxx val val xxx val val Key colB colC xxx val val xxx val val Key colB colC xxx val val xxx val val
  118. 118. © 2017 MapR Technologies HBase Schema With Hbase/MapR-DB data is automatically partitioned by Key Range
  119. 119. © 2017 MapR Technologies Spark Streaming writing to MapR-DB (HBase API)
  120. 120. © 2017 MapR Technologies Spark HBase and MapR-DB Binary Connector •  HConnection object in every Spark Executor: •  allowing for distributed parallel writes, reads, or scans
  121. 121. © 2017 MapR Technologies Spark Hbase streamBulkPut •  HBaseContext streamBulkPut method parameters: •  message value DStream, the TableName to write to, function to convert the Dstream values to HBase put records.
  122. 122. © 2017 MapR Technologies Massively Parrallel writes to HBase The Spark Streaming bulk put enables massively parallel sending of puts to HBase
  123. 123. © 2017 MapR Technologies HBase Schema To use the Spark HBase Connector for reads, you need to define the Catalog for the schema mapping between the HBase and Spark
  124. 124. © 2017 MapR Technologies SparkSQL and DataFrames: Define the Schema define the Catalog for the schema mapping between the HBase and Spark
  125. 125. © 2017 MapR Technologies Loading data from MapR-DB into a Spark DataFrame Use Catalog defining schema
  126. 126. © 2017 MapR Technologies Spark Dataframes combine filters and select filters rows for cluster ids (the beginning of the row key) >= 9. The select selects a set of columns: key, lat, and lon.
  127. 127. © 2017 MapR Technologies Use Case: Real-Time Data Pipelines Input Data + Actual Delay Input Data + Predictions Consumer withML Model 2 Consumer withML Model 1 Decoy results Consumer Consumer withML Model 3 Consumer Stream Archive Stream Scores Stream Input SQL SQL Real time Flight Data Stream Input Actual Delay Input Data + Predictions + Actual Delay Real Time dashboard + Historical Analysis
  128. 128. © 2017 MapR Technologies
  129. 129. © 2017 MapR Technologies To Learn More: •  MapR Free ODT http://learn.mapr.com/
  130. 130. © 2017 MapR Technologies MapR Blog • https://www.mapr.com/blog/
  131. 131. © 2017 MapR Technologies MapR Container for Developers • https://maprdocs.mapr.com/home/MapRContainerDevelopers/ MapRContainerDevelopersOverview.html
  132. 132. © 2017 MapR Technologies …helping you put data technology to work ●  Find answers ●  Ask technical questions ●  Join on-demand training course discussions ●  Follow release announcements ●  Share and vote on product ideas ●  Find Meetup and event listings Connect with fellow Apache Hadoop and Spark professionals community.mapr.com
  133. 133. © 2017 MapR Technologies Stream Processing Building a Complete Data Architecture MapR File System (MapR-XD) MapR Converged Data Platform MapR Database (MapR-DB) MapR Event Streams Sources/Apps Bulk Processing
  134. 134. © 2017 MapR Technologies Q&A ENGAGE WITH US
  • abhirajgupta3

    Oct. 24, 2019
  • ssuserce170b

    Jun. 15, 2019
  • HYDN

    Sep. 20, 2018
  • GuRkhali

    Apr. 18, 2018
  • StreamingAnalytics

    Apr. 17, 2018

Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using Apache APIs: Kafka, Spark, HBase

Views

Total views

2,027

On Slideshare

0

From embeds

0

Number of embeds

6

Actions

Downloads

75

Shares

0

Comments

0

Likes

5

×