6. Spark Streaming
• First attempt at unifying streaming and batch
• State management built in
• Exactly once semantics
• Features required for large clusters
• Straggler mitigation,dynamic load balancing,fast fault-recovery
12. Processing
Businesslogic change & new ops
(windows,sessions)
Complex Programming Models
Output
How do we define
outputover time & correctness?
Data
Late arrival, varying distribution overtime, …
16. Structured Streaming
High-level streaming API built on SparkSQL engine
• Runsthe same querieson DataFrames
• Eventtime, windowing,sessions,sources& sinks
Unifies streaming, interactive and batch queries
• Aggregate data in a stream, then serve using JDBC
• Change queriesatruntime
• Build and apply ML models
17. output for
data at 1
Result
Query
Time
data up
to PT 1
Input
complete
output
Output
1 2 3
Trigger: every 1 sec
data up
to PT 2
output for
data at 2
data up
to PT 3
output for
data at 3
Model
18. delta
output
output for
data at 1
Result
Query
Time
data up
to PT 2
data up
to PT 3
data up
to PT 1
Input
output for
data at 2
output for
data at 3
Output
1 2 3
Trigger: every 1 sec
Model
20. Example: ETL
Input: files in S3
Query: map (transform each record)
Trigger: “every5 sec”
Output mode: “newrecords”,into S3 sink
21. Example: Page View Count
Input: recordsin Kafka
Query: select count(*) group by page, minute(evtime)
Trigger: “every5 sec”
Output mode: “update-in-place”, into MySQL sink
Note: this will automatically update “old” recordson late data!
22. Logically:
DataFrame operations on static data
(i.e. as easyto understand as batch)
Physically:
Spark automatically runs the queryin
streaming fashion
(i.e. incrementally and continuously)
DataFrame
Logical Plan
Continuous,
incremental execution
Catalyst optimizer
Execution
26. Rest of Spark will follow
• Interactive queriesshould just work
• Spark’s data sourceAPI will be updated to support seamless
streaming integration
• Exactly once semantics end-to-end
• Different outputmodes (complete,delta, update-in-place)
• ML algorithms will be updated too
27. What can we do with this that’s hard
with other engines?
Ad-hoc, interactive queries
Dynamic changing queries
Benefits of Spark: elastic scaling, stragglermitigation, etc
28. Use Case: Fraud Detection
STREAM
ANOMALY
Machine LearningModel
continuously updates
to detectnew anomalies
Analyze Historic Data
29. Timeline
Spark 2.0
• API foundation
• Kafka, file systems, and
databases
• Event-time aggregations
Spark 2.1 +
• Continuous SQL
• BI app integration
• Other streaming sources/ sinks
• Machine learning