Real-Time Analysis of Popular Uber Locations using Apache APIs: Machine Learning, Spark Streaming, Kafka with MapR-ES and MapR-DB

Real-Time Analysis of Popular Uber Locations
using Apache APIs:
•  Spark Machine Learning,
•  Spark Structured Streaming,
•  Kafka
with MapR-ES and MapR-DB

2 © 2018 MapR Technologies, Inc
•  Overview of Unsupervised Machine Learning Clustering
•  Use K-Means to Cluster Uber locations and save ML model
•  Overview of Kafka API
•  Use Spark Structured Streaming:
•  To Read from Kafka topic
•  Enrich with ML model
•  Write to MapR-DB JSON document database
•  Use Spark SQL to query MapR-DB database

Agenda
2

Use Case: Real-Time Analysis of Geographically Clustered Vehicles

What is Machine Learning?
Data Build ModelTrain Algorithm
Finds patterns
New Data Use Model
(prediction function)
Predictions
Contains patterns Recognizes patterns

ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set
Evaluate Results
Historical
Data
Deployed
Model
Insights
Data
Discovery,
Model
Creation
Production
Feature Extraction
Feature
Extraction
Uber
trips
Stream
TopicUber
trips
New Data

Supervised and Unsupervised Machine Learning
Machine Learning
Unsupervised
•  Clustering
•  Collaborative Filtering
•  Frequent Pattern Mining
Supervised
•  Classification
•  Regression
Label

Supervised Algorithms use labeled data
Data
features
Build Model
New Data
features
Predict
Use Model
X1, X2
Y
f(X1, X2) =Y
X1, X2
Y

Unsupervised Algorithms use Unlabeled data
Customer GroupsBuild ModelTrain Algorithm
Finds patterns
New Customer
Purchase Data
Use Model
Similar Customer Group
Contains patterns Recognizes patterns
Customer purchase
data

Unsupervised Machine Learning: Clustering
Clustering
group news articles into different categories

Clustering: Definition
Unsupervised learning task
Groups objects into clusters of high similarity

Clustering: Definition
Unsupervised learning task
Groups objects into clusters of high similarity
–  Search results grouping
–  Grouping of customers, patients
–  Text categorization
–  recommendations
•  Anomaly detection: find what’s not similar

Clustering: Example
Group similar objects

Clustering: Example
Use MLlib K-means algorithm
1.  Initialize coordinates to K cluster centers

Clustering: Example
1.  Initialize coordinates to K clusters
centers (centroid)
2.  Assign all points to nearest cluster
center (centroid)

Clustering: Example
1.  Initialize coordinates to center of clusters
(centroid)
2.  Assign all points to nearest centroid
3.  Update centroids to center of assigned
points

Clustering: Example
1.  Initialize coordinates to center of clusters
(centroid)
2.  Assign all points to nearest centroid
3.  Update centroids to center of points
4.  Repeat until conditions met

How a Spark Application Runs on a Cluster

Spark Distributed Datasets
partitioned
•  Read only collection of typed objects
Dataset[T]
•  Partitioned across a cluster
•  Operated on in parallel
•  in memory can be Cached

Loading a Dataset

Dataset Read From a File
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
tasks
tasks
tasks

Dataset Read From a File
Worker
Worker
Worker
Block 1
Block 2
Block 3
Driver
Cache 1
Cache 2
Cache 3
Process
& Cache
Data
Process
& Cache
Data
Process
& Cache
Data

Date/Time: The date and time of the Uber pickup
Lat: The latitude of the Uber pickup
Lon: The longitude of the Uber pickup
Base: The TLC base company affiliated with the Uber pickup

The Data Records are in CSV format. An example line is shown below:
2014-08-01 00:00:00,40.729,-73.9422,B02598

Uber Data

case class Uber(dt: String, lat: Double, lon: Double, base: String)

val schema = StructType(Array(
StructField("dt", TimestampType, true),
StructField("lat", DoubleType, true),
StructField("lon", DoubleType, true),
StructField("base", StringType, true)
))

Load the data into a Dataframe: Define the Schema

val df = spark.read.format("csv").option("inferSchema", "false")
.schema(schema).option("header", "false")
.load(file)
Load the data into a Dataframe

Load the data into a DataFrame
columns
row

val df = spark.read.format("csv").option("inferSchema", "false")
.schema(schema).option("header", "false")
.load(file).as[Uber]

Load the data into a Dataset

Load the data into a Dataset
Collection of Uber
objects
columns
row

•  in Spark 2.0, DataFrame APIs merged with Datasets APIs
•  A Dataset is a collection of typed objects (SQL and functions)
•  Dataset[T]
•  A DataFrame is a Dataset of generic Row objects (SQL)
•  Dataset[Row]

Dataset merged with Dataframe

Spark Distributed Datasets
Transformations create a new Dataset
from the current one,
Lazily evaluated
Actions return a value to the driver

Spark ML workflow

Feature Vectors are vectors of numbers representing the value for each feature

Extract the Features
Image reference O’Reilly Learning Spark
+
+ ̶+
̶ ̶
Feature Vectors Model
Featurization Training
Model
Evaluation
Best Model
Training Data
+
+
̶+
̶ ̶
+
+
̶+
̶ ̶
+
+
̶+
̶ ̶
+
+ ̶+
̶ ̶

Uber Example
•  What are the “if questions” or properties
we can use to group?
–  These are the Features:
–  We will group by Lattitude, longitude
•  Use Spark SQL to analyze: Day of the
week, time, rush hour for groups …
•  NOTE: this example uses real Uber data,
but the code is from me, not Uber
NEAR REALTIME
PRICE SURGING

val featureCols = Array("lat", "lon")
val assembler = new VectorAssembler()
.setInputCols(featureCols)
.setOutputCol("features")
val df2 = assembler.transform(df)

Use VectorAssembler to put features in vector column

val kmeans = new KMeans()
.setK(10)
.setFeaturesCol("features")
.setPredictionCol("cid")
.setMaxIter(20)

Create Kmeans Estimator, Set Features

val model = kmeans.fit(df2)

Fit the Model on the Training Data Features

model.clusterCenters.foreach(println)

[40.76930621976264,-73.96034885367698]
[40.67562793272868,-73.79810579052476]
[40.68848772848041,-73.9634449047477]
[40.78957777777776,-73.14270740740741]
[40.32418330308531,-74.18665245009073]
[40.732808848486286,-74.00150153727878]
[40.75396549974632,-73.57692359208531]
[40.901700842900674,-73.868760398198]

Cluster Centers from fitted model

Clusters from fitted model

K-means
model
val clusters = model.summary.predictions
Or
val clusters = model.transform(df3)
clusters.createOrReplaceTempView("uber”)
clusters.show()

Analyze Clusters
summary DataFrame +
Features +
cluster

clusters.groupBy("cid").count().orderBy(desc( "count")).show(5)
+---+-----+
|cid|count|
+---+-----+
| 6|83505|
| 5|79472|
| 0|56241|
| 16|26933|
| 13|23581|
+---+-----+

Which clusters had the highest number of pickups?

Which clusters had the highest number of pickups?
%sql
SELECT COUNT(cid), cid
FROM uber
GROUP BY cid
ORDER BY COUNT(cid) DESC

How many pickups occurred in the busiest 5 clusters by hour?
select hour(uber.dt) as hr,cid, count(cid) as ct
from uber where cid in (0,8,9,13,17)
group By hour(uber.dt), cid

Which hours had the highest number of pickups?
SELECT hour(uber.dt) as hr,count(cid) as ct
FROM uber
GROUP BY hour(uber.dt)

fitted
model
model.write.overwrite().save("/path/savemodel")

Use later

val sameModel = KMeansModel.load("/user/user01/data/savemodel")

Save the model to distributed file system
saveDataFrame +
Features

hadoop fs -ls /user/mapr/ubermodel/metadata
/user/mapr/ubermodel/metadata/_SUCCESS
/user/mapr/ubermodel/metadata/part-00000
hadoop fs -ls /user/mapr/ubermodel/data
/user/mapr/ubermodel/data/_SUCCESS
/user/mapr/ubermodel/data/part-00000-4d20b313-ddc1-43cb-
a863-434a36330639-c000.snappy.parquet
hadoop fs -cat /user/mapr/ubermodel/metadata/part-00000
{"class":"org.apache.spark.ml.clustering.KMeansModel","timestamp":
1540826934502,"sparkVersion":"2.3.1-mapr-1808","uid":"kmeans_4ad427355253","paramMap":
{"predictionCol":"cid","seed":1,"initMode":"k-means||","featuresCol":"features","initSteps":
2,"maxIter":100,"tol":1.0E-4,"k":20}}
The model on the distributed file system

What is a Stream ?
•  A stream is an continuous sequence of events or records
•  Records are key-value pairs

Examples of Streaming Data
Fraud detection Smart Machinery Smart Meters Home Automation
Networks Manufacturing Security Systems Patient Monitoring

A Stanford team has shown that a machine-learning model can identify arrhythmias
from an EKG better than an expert
•  https://www.technologyreview.com/s/608234/the-machines-are-getting-ready-
to-play-doctor/
Example of Streaming Data combined with Machine Learning

https://mapr.com/blog/ml-iot-connected-medical-devices/
Applying Machine Learning to Live Patient Data

Collect the Data
Data IngestSource
Stream
Topic
•  Data Ingest:
–  Using the Kafka API

Topics:
Logical collection of events
Organize Events into Categories
Organize Data into Topics with the MapR Event Store for Kafka
Consumers
MapR Cluster
Topic: Pressure
Topic: Temperature
Topic: Warnings
Consumers
Consumers
Kafka API Kafka API

Topics are partitioned for throughput and
scalability

Scalable Messaging with MapR Event Streams
Server 1
Partition1: Topic - Pressure
Partition1: Topic - Temperature
Partition1: Topic - Warning
Server 2
Server 3

Producers are load balanced
between partitions
Kafka API

Consumers
Consumers
Consumers
Consumer
groups can
read in parallel
Kafka API

New Messages are
Added to the end
Partition is like an Event Log
New
Message
6 5 4 3 2 1
Old
Message

Messages are delivered in the order they are received
Partition is like a Queue

Messages remain on the partition, available to other consumers

Unlike a queue, events are still persisted after they’re delivered

Messages can be persisted forever
Or
Older messages can be deleted automatically based on time to live

When Are Messages Deleted?
MapR Cluster
6 5 4 3 2 1Partition
1
Older
message

How do we do this with High Performance at Scale?
•  Parallel operations
•  minimizes disk read/writes

Processing Same Message for Different Purposes

Process the Data with Spark Structured Streaming

Datasets Read from Stream
Task
Cache
Process
& Cache
Data
offsets
Stream
partition
Task
Cache
Process
& Cache
Data
Task
Cache
Process
& Cache
Data
Driver
Stream
partition
Stream
partition
Data is cached for
aggregations
And windowed
functions

new data in the
data stream
=
new rows appended
to an unbounded table
Data stream as an unbounded table

Treat Stream as Unbounded Tables

The Stream is continuously processed

Spark automatically streamifies SQL plans
Image reference Databricks

Stream Processing

ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set
Evaluate Results
Historical
Data
Deployed
Model
Insights
Data
Discovery,
Model
Creation
Production
Feature Extraction
Feature
Extraction
Uber
trips
Stream
TopicUber
trips
New Data

// load the saved model from the distributed file system
val model = KMeansModel.load(modelpath)
Load the saved model

val df1 = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "maprdemo:9092")
.option("subscribe", "/apps/uberstream:ubers”)
.option("startingOffsets", "earliest")
.option("failOnDataLoss", false)
.option("maxOffsetsPerTrigger", 1000)
.load()

Streaming pipeline Kafka Data source

case class Uber(dt: String, lat: Double, lon: Double, base: String,
rdt: String)

// Parse string into Uber case class
def parseUber(str: String): Uber = {
val p = str.split(",")
Uber(p(0), p(1).toDouble, p(2).toDouble, p(3), p(4))
}

Function to Parse CSV data to Uber Object

//register a user-defined function (UDF) to deserialize the message
spark.udf.register("deserialize",
(message: String) => parseUber(message))
//use the UDF in a select expression
val df2 = df1.selectExpr("""deserialize(CAST(value as STRING)) AS
message""").select($"message".as[Uber])

Parse message txt to Uber Object

val featureCols = Array("lat", "lon”)
val assembler = new VectorAssembler()
.setInputCols(featureCols)
.setOutputCol("features")

val df3 = assembler.transform(df2)

Use VectorAssembler to put Features in a column

//use model to get the cluster ids from the features
val clusters1 = model.transform(df3)

Use Model to get Cluster Ids from the features

//select columns we want to keep
val clusters= clusters1.select($"dt".cast(TimestampType),
$"lat", $"lon", $"base",$"rdt", $”cid”)
// Create object with unique Id for mapr-db
case class UberwId(_id: String, dt: java.sql.Timestamp, base: String, cid: Integer,
clat: Double, clon: Double)
val cdf = clusters.withColumn("_id", concat($"cid", lit("_"), $"rdt")).as[UberwId]
// cdf is like this:
+--------------------+-------------------+-------+--------+------+---+----------------+------------------+
| _id| dt| lat| lon| base|cid| clat| clon|
+--------------------+-------------------+-------+--------+------+---+----------------+------------------+
|0_922337049642672...|2014-08-18 08:36:00| 40.723|-74.0021|B02598| 0|40.7173662333218|-74.00933866774037|
|0_922337049642672...|2014-08-18 08:36:00|40.7288|-74.0113|B02598| 0|40.7173662333218|-74.00933866774037|
|0_922337049642672...|2014-08-18 08:35:00|40.7417|-74.0488|B02617| 0|40.7173662333218|-74.00933866774037|
Create Unique Id for MapR-DB row key

Writing to a Memory Sink
Write results to MapR-DB
Start running the query
val query = cdf.writeStream
.format(MapRDBSourceConfig.Format)
.option(MapRDBSourceConfig.TablePathOption, tableName)
.option(MapRDBSourceConfig.IdFieldPathOption, "_id")
.option(MapRDBSourceConfig.CreateTableOption, false)
.option("checkpointLocation", "/user/mapr/ubercheck")
.option(MapRDBSourceConfig.BulkModeOption, true)
.option(MapRDBSourceConfig.SampleSizeOption, 1000)

query.start().awaitTermination()

%sql select * from uber limit 3:
Streaming Applicaton

SELECT hour(uber.dt) as hr,cid, count(cid) as ct FROM uber group By hour(uber.dt),
cid

Stream Processing Pipeline

MapR-DB Connector for Apache Spark
Spark Streaming writing to MapR-DB JSON

Spark MapR-DB Connector

Relational Database vs. MapR-DB
bottleneck
Storage ModelRDBMS MapR-DB
Normalized schema à Joins for
queries can cause bottleneck De-Normalized schema à Data that
is read together is stored together
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val

Designed for Partitioning and Scaling

MapR-DB JSON Document Store
Data is automatically partitioned
and sorted by _id row key!

Writing to a MapR-DB Sink
Write Streaming DataFrame
Query Results to MapR-DB

Start running the query
val query = cdf.writeStream
.format(MapRDBSourceConfig.Format)
.option(MapRDBSourceConfig.TablePathOption, tableName)
.option(MapRDBSourceConfig.IdFieldPathOption, "_id")
.option(MapRDBSourceConfig.CreateTableOption, false)
.option("checkpointLocation", "/user/mapr/ubercheck")
.option(MapRDBSourceConfig.BulkModeOption, true)
.option(MapRDBSourceConfig.SampleSizeOption, 1000)

query.start().awaitTermination()

Explore the Data With Spark SQL

•  Spark SQL queries and updates to MapR-DB
•  With projection and filter pushdown, custom partitioning, and data locality

Spark SQL Querying MapR-DB JSON

val df: Dataset[UberwId] = spark
.loadFromMapRDB[UberwId](tableName, schema)
.as[UberwId]

Spark Distributed Datasets read from MapR-DB Partitions
Worker
Task
Worker
Driver
Cache 1
Cache 2
Cache 3
Process
& Cache
Data
Process
& Cache
Data
Process
& Cache
Data
Task
Task
Driver
tasks
tasks
tasks

Data
Frame
Load data
df.createOrReplaceTempView("uber")
df.show

Load the data into a Dataframe
Data is automatically partitioned
and sorted by _id row key!

val res = df.groupBy(“cid")
.count()
.orderBy(desc(count))
.show(5)
+---+------+
|cid| count|
+---+------+
| 6|197225|
| 5|192073|
| 0|131296|
| 16| 62465|
| 13| 52408|
+---+------+
Top 5 Cluster trip counts ?

val points = df.select("lat","lon”,"cid”).orderBy(desc("dt"))
Display latest locations and Cluster centers on a Google Map

df.filter($"_id" <= ”1”).select(hour($"dt").alias("hour"), $"cid")
.groupBy("hour","cid").agg(count("cid")
.alias("count"))
Which hours have the highest pickups for cluster id 0 ?

df.filter($"_id" <= "1").select(hour($"dt").alias("hour"), $"cid")
.groupBy("hour","cid").agg(count("cid").alias("count"))
.orderBy(desc( "count")).explain
== Physical Plan ==
*(3) Sort [count#120L DESC NULLS LAST], true, 0
+- Exchange rangepartitioning(count#120L DESC NULLS LAST, 200)
+- *(2) HashAggregate(keys=[hour#113, cid#5], functions=[count(cid#5)])
+- Exchange hashpartitioning(hour#113, cid#5, 200)
+- *(1) HashAggregate(keys=[hour#113, cid#5],
functions=[partial_count(cid#5)])
+- *(1) Project [hour(dt#1, Some(Etc/UTC)) AS hour#113, cid#5]
+- *(1) Filter (isnotnull(_id#0) && (_id#0 <= 1))
+- *(1) Scan MapRDBRelation(/user/mapr/ubertable
[dt#1,cid#5,_id#0]
PushedFilters: [IsNotNull(_id), LessThanOrEqual(_id,1)]
MapR-DB Projection and Filter push down

Spark MapR-DB Projection Filter push down
Projection and Filter pushdown reduces the
amount of data passed between MapR-DB
and the Spark engine when selecting and
filtering data.

Data is selected and filtered in
MapR-DB!

SELECT hour(uber.dt) as hr,cid, count(cid) as ct FROM uber
GROUP BY hour(uber.dt), cid
Which hours and Clusters have the highest pick ups?

MapR Data Platform

Link to Code for this webinar is in
appendix of this book.
https://mapr.com/ebook/getting-started-
with-apache-spark-v2/
New Spark Ebook

MapR Free ODT http://learn.mapr.com/
To Learn More: New Spark 2.0 training

https://mapr.com/blog/
MapR Blog

Real-Time Analysis of Popular Uber Locations using Apache APIs: Machine Learning, Spark Streaming, Kafka with MapR-ES and MapR-DB

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Real-Time Analysis of Popular Uber Locations using Apache APIs: Machine Learning, Spark Streaming, Kafka with MapR-ES and MapR-DB

Similar to Real-Time Analysis of Popular Uber Locations using Apache APIs: Machine Learning, Spark Streaming, Kafka with MapR-ES and MapR-DB (20)

More from Carol McDonald

More from Carol McDonald (13)

Recently uploaded

Recently uploaded (20)

Real-Time Analysis of Popular Uber Locations using Apache APIs: Machine Learning, Spark Streaming, Kafka with MapR-ES and MapR-DB