Introduction to Spark Structured Streaming

1. Our product is a transformational product that uses third
generation Big Data technologies to execute the most
comprehensive form of Digital Transformation
INTRODUCING
SPARK STRUCTURED
STREAMING

2. José Carlos García Serrano
Big Data Architect in Stratio.
I am from Granada and computer science engineer in
the ETSII, post graduate in Big Data and certificate in
Spark and AWS
def fanBoy(): Seq[Skills] = {
val functional = Seq(Scala, Akka)
val processing = Seq(Spark)
val noSql = Seq(MongoDB, Cassandra)
functional ++ processing ++ noSql
}
def aLongTimeAgo(): Seq[Skills] = {
val programming = Seq(Delphi, C++)
val processing = Seq(Hadoop)
val sql = Seq(Interbase, FireBird)
programming ++ processing ++ sql
}

3. INDEXINDEX
1. Introduction
2. Basic concepts
3. Continuous aggregations
4. Stateful operations
5. Kafka integration
6. Benchmark and other streaming frameworks

4. Introduction
1

5. One Streaming API to rule
them all
Spark Core + Spark SQL + Spark Streaming =
Spark Structured Streaming

6. Introduction

7. Introduction
Main features
● Integrated in SQL API
● Easy to use
● Continuous processing with exactly one
● Interactive queries
● Joins with static data integrating Streaming and Batch
● Continuous aggregations
● Stateful operations
● Low latency (<1ms)

8. Introduction

9. Basic concepts
2

10. Basic concepts
Load from Stream Sources
val csvDF = spark
.readStream
.option("sep", ";")
.schema(userSchema)
.csv("/path/to/directory")
.load()
val kafkaDF = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1")
.option("subscribe", "topic1")
.load()
Kafka, Files(json, csv, parquet, txt), Socket and more ….

11. Basic concepts
Output Sinks
csvDF.writeStream
.format("parquet")
.option("path", "path/to/destination/dir")
.start()
csvDF.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1")
.option("topic", "topic2")
.start()
Kafka, Files(json, csv, parquet, txt), JDBC, Foreach, Memory and more ….

12. Basic concepts
Triggers
csvDF.writeStream
.format("parquet")
.trigger(Trigger.Once())
.option("path", "path/to/destination/dir")
.start()
csvDF.writeStream
.format("kafka")
.trigger(Trigger.ProcessingTime(6,
seconds))
.option("kafka.bootstrap.servers", "host1:port1")
.option("topic", "topic2")
.start()
Continuous triggering, one execution or periodical execution

13. Basic concepts

14. Basic concepts
Integration with Spark SQL and Dataset API
val dataFrameStream : DataFrame = …….
dataFrameStream.createOrReplaceTempView(“streamTable”)
sparkSession.sql(“select * from streamTable”)
Catalyst and Tungsten optimizations!!!

15. Basic concepts
Mixing Streaming data with Batch data
val streamDf : DataFrame = spark.readStream
val staticDf : DataFrame = spark.read
streamDf.join(staticDf, “type”)
Two worlds in a line of code!!!
DStream.foreachRDD(rdd =>
val staticDf = spark.sql(“select …...”)
val streamDf = spark.createDataFrame(rdd,
schema)
streamDf.join(staticDf)
)

16. Basic concepts
Data cleaning (drop duplicates)
val dataFrameStream : DataFrame = …….
dataFrameStream.dropDuplicates("idColumn")
Optimized in DataFrames with watermark, without watermark all the
past events has stored in the State

17. Basic concepts
Output modes
● Complete -> The entire updated Result Table will be written
(only with aggregations)
● Append -> Only the new rows appended in the Result Table will be written
(with no aggregations, joins, with aggregations, with watermark)
● Update -> Only the rows that were updated in the Result Table will be written
(with no aggregations, with aggregations, no agg with watermark)

18. Basic concepts
Exactly one
● Source must manage offsets (Kafka or Kinesis)
● Checkpointing and write ahead logs
● Sink idempotent (transactional updates available in file system sink)

19. Basic concepts

20. Continuous
aggregations
3

21. Continuous aggregations
SQL:
select count(*) from words group by word
Spark Dataset:
words.groupBy("word").count()
Minimal state storage!!

22. Continuous aggregations
Incremental queries with windowed aggregations

23. Continuous aggregations
Windowed aggregation example
val words : DataFrame = …..
words.groupBy(
window(
timeColumn = $"timestamp",
windowDuration = "10 minutes",
slideDuration = "5 minutes"
),$"word"
).count()
6 lines of code in Scala!!!

24. Now we know how to build
continuous and incremental
aggregations
We need to optimize this aggregations for production
deployments

25. Continuous aggregations
With the watermarks Spark expire late data

26. Continuous aggregations

27. Stateful operations
4

28. Stateful operations

29. Stateful operations
Checkpointing integrated
csvDF.writeStream
.format("parquet")
.option("checkpointLocation", "/tmp/checkpoint")
.option("path", "path/to/destination/dir")
.start()

30. Stateful operations
Easy API with all functionalities
csvDF.groupByKey(...)
.mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout)(mappingFunction)
def mappingFunction(key: K, value: Iterator[V], state: GroupState[S]): String = {
if (state.hasTimedOut) { // If called when timing out, remove the state
state.remove()
} else if (state.exists) { // If state exists, use it for processing
val existingState = state.get // Get the existing state
val shouldRemove = ... // Decide whether to remove the state
if (shouldRemove) {
state.remove() // Remove the state
} else {
val newState = ...
state.update(newState) // Set the new state
state.setTimeoutDuration("1 hour") // Set the timeout
}
} else {
val initialState = ...
state.update(initialState) // Set the initial state
state.setTimeoutDuration("1 hour") // Set the timeout
}
}

31. “
If you want to build complex
aggregations you should use this
API or state functions in
Streaming API

32. Kafka integration
5

33. Spark Streaming and
Structured Streaming should
be used with Kafka
Integrated as a Source and as a Sink!!
It’s available now in batch

34. Kafka integration

35. Kafka integration
Reading streaming data
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1")
.load()

36. Kafka integration
Reading streaming data for batch jobs
spark.read
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1,topic2")
.option("startingOffsets", """{"topic1":{"0":23,"1":-2},"topic2":{"0":-2}}""")
.option("endingOffsets", """{"topic1":{"0":50,"1":-1},"topic2":{"0":-1}}""")
.load()
…
.option("subscribePattern", "topic.*")
.option("startingOffsets", "earliest")
.option("endingOffsets", "latest")

37. Kafka integration
Sink streaming data
spark.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("topic", "topic1")
.start()

38. Benchmark and other
streaming frameworks
6

39. Benchmark and other streaming frameworks

40. Benchmark and other streaming frameworks

41. Benchmark and other streaming frameworks
All that glitters is not
gold!!!

42. Contact me and check some examples in my GitHub:
● gserranojc@gmail.com
● www.linkedin.com/in/gserranojc
● https://github.com/compae/structured-streaming-tests
Spark documentation:
● https://spark.apache.org/docs/latest/structured-streaming-
programming-guide.html