In Spark 2.0, we introduced Structured Streaming, which allows users to continually and incrementally update your view of the world as new data arrives, while still using the same familiar Spark SQL abstractions. I talk about progress we’ve made since then on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications.

1. Insights Without
Tradeoffs Using
Structured Streaming
Michael Armbrust - @michaelarmbrust
Spark Summit East 2017

2. Parallelism
Split up the problem
to harness many
machines for computation
Complexity
Handle cross-machine
communication and failures

3. Developer Productivity
Quickly and concisely express
common computations
Efficiency
Hand-tune code to minimize
overheads and process the
most data per cycle
SQL

4. Throughput
Process large historical
repositories quickly
Latency
Up-to-date answers
as new data arrives
Structured Streaming

5. Production Use Cases

6. Streaming at
Collect logs and metrics from a variety of
sources to ensure the security,
availability and performance of our cloud
platform.

7. Engineer Office Hours
MY HOURS
• Spark Core
• R
• Data Science
• ML
• GraphFrames, Deep Learning
• Databricks
• Spark SQL
• Structured Streaming
• Spark SQL
• Structured Streaming
• Databricks
TODAY 4:30 –
5:15
OTHER ENGINEERS
TODAY 1:45 –
5:15
THURS 10:30 –
2:30
@ Databricks Booth

8. Thank you!
All code available at databricks.com/blog
and @michaelarmbrust