Fast Data architectures are the answer to the increasing need for the enterprise to process and analyze continuous streams of data to accelerate decision making and become reactive to the particular characteristics of their market. Apache Spark is a popular framework for data analytics. Its capabilities include SQL-based analytics, dataflow processing, graph analytics and a rich library of built-in machine learning algorithms. These libraries can be combined to address a wide range of requirements for large-scale data analytics. To address Fast Data flows, Spark offers two API's: The mature Spark Streaming and its younger sibling, Structured Streaming. In this talk, we are going to introduce both APIs. Using practical examples, you will get a taste of each one and obtain guidance on how to choose the right one for your application.

2. Gerard Maas
Señor SW Engineer
Computer Engineer
Scala Developer
Early Spark Adopter (v0.9)
Cassandra MVP (2015, 2016)
Stack Overflow Top
Contributor
(Spark, Spark Streaming, Scala)
Wannabe {
IoT Maker
Drone crasher/tinkerer
}
@maasg
https://github.com/maasg
https://www.linkedin.com/
in/gerardmaas/
https://stackoverflow.com
/users/764040/maasg

4. Streaming | Big Data

5. 100Tb
5Mb

6. 100Tb
5Mb/s

7. ∑ Stream = Dataset
Dataset = Stream

8. What is Spark and Why we Should Care
Streaming APIs in Spark
- Structured Streaming Overview
- Interactive Session 1
- Spark Streaming Overview
- Interactive Session 2
Spark Streaming [AND|OR|XOR] Structured Streaming

9. Once upon a
time...

10. Apache Spark Core
Spark SQL
SparkMLLib
SparkStreaming
Structured
Streaming
DataFrames
DataSets
GraphFrames
Data Sources

11. Apache Spark Core
Spark SQL
SparkMLLib
SparkStreaming
Structured
Streaming
DataFrames
DataSets
GraphFrames
Data Sources

12. 1Structured
Streaming

13. Structured Streaming
Kafka
Sockets
HDFS/S3
Custom
Streaming
DataFrame
Query
Kafka
Files
foreachSink
console
memory
Output
Mode

14. Sensor
Data
Producer
Structured
Streaming
Fast Data Platform
Spark Notebook
Local Process
Demo Scenario

15. 1 Structured Streaming
HIGHLIGHTS

16. val rawData = sparkSession.readStream
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServer)
.option("subscribe", sourceTopic)
.option("startingOffsets", "latest")
.load()
Sources

17. Operations
...
val rawValues = rawData.selectExpr("CAST(value AS STRING)").as[String]
val jsonValues = rawValues.select(from_json($"value", schema) as "record")
val sensorData = jsonValues.select("record.*").as[SensorData]
…

18. Event Time
...
val movingAverage = sensorData.withColumn("timestamp", toSeconds($"ts").cast(TimestampType))
.withWatermark("timestamp", "30 seconds")
.groupBy($"id", window($"timestamp", "30 seconds", "10 seconds"))
.agg(avg($"temp"))
...

19. Sinks...
val visualizationQuery = sensorData.writeStream
.queryName("visualization") // this query name will be the SQL table name
.outputMode("update")
.format("memory")
.start()
...
val kafkaWriterQuery = kafkaFormat.writeStream
.queryName("kafkaWriter") // this query name will be the table name
.outputMode("append")
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServer)
.option("topic", targetTopic)
.option("checkpointLocation", "/tmp/spark/checkpoint")
.start()

20. Use Cases
● Streaming ETL
● Stream aggregations, windows
● Event-time oriented analytics
● Join Streams with Fixed Datasets
● Apply Machine Learning Models

21. 2Spark
Streaming

22. Spark Streaming
Kafka
Flume
Kinesis
Twitter
Sockets
HDFS/S3
Custom Apache Spark
SparkSQL
SparkML
...
Databases
HDFS
API Server
Streams

23. DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
t0 t1 t2 t3 ti ti+1
RDD[U] RDD[U] RDD[U] RDD[U] RDD[U]
Transformation
T -> U
Actions

24. API: Transformations
map,
flatmap,
filter
count,
reduce,
countByValue,
reduceByKey
n
union,
join
cogroup

25. API: Transformations
mapWithState
…
…

26. API: Transformations
transform
val iotDstream = MQTTUtils.createStream(...)
val devicePriority = sparkContext.cassandraTable(...)
val prioritizedDStream = iotDstream.transform{rdd =>
rdd.map(d => (d.id, d)).join(devicePriority)
}

27. Actions
print
-------------------------------------------
Time: 1459875469000 ms
-------------------------------------------
data1
data2
saveAsTextFiles,
saveAsObjectFiles,
saveAsHadoopFiles
xxx
yyy
zzz
foreachRDD
*

28. Actions
print
-------------------------------------------
Time: 1459875469000 ms
-------------------------------------------
data1
data2
saveAsTextFiles,
saveAsObjectFiles,
saveAsHadoopFiles
xxx
yyy
zzz
foreachRDD
*
Spark SQL
Dataframes
GraphFrames
Any API

29. Sensor
Data
Producer
Structured
Streaming
Fast Data Platform
Spark Notebook
Local Process
Demo Scenario

30. 2 Spark Streaming
HIGHLIGHTS

31. import org.apache.spark.streaming.StreamingContext
val streamingContext = new StreamingContext(sparkContext, interval)
Streaming Context

32. val kafkaParams = Map[String, String](
"metadata.broker.list" -> kafkaBootstrapServer,
"group.id" -> "sensor-tracker-group",
"auto.offset.reset" -> "largest",
"enable.auto.commit" -> (false: java.lang.Boolean).toString
)
val topics = Set(topic)
@transient val stream = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](
streamingContext, kafkaParams, topics)
Source

33. import spark.implicits._
val sensorDataStream = stream.transform{rdd =>
val jsonData = rdd.map{case (k,v) => v}
val ds = sparkSession.createDataset(jsonData)
val jsonDF = spark.read.json(ds)
val sensorDataDS = jsonDF.as[SensorData]
sensorDataDS.rdd
}
Transformations

34. val model = new M2Model()
…
model.trainOn(inputData)
…
val scoredDStream = model.predictOnValues(inputData)
Model

35. suspects.foreachRDD{rdd =>
val sample = rdd.take(20).map(_.toString)
val total = s"total found: ${rdd.count}"
outputBox(total +: sample)
}
Output

36. Usecases
● Stream-stream joins
● Complex state management (local + cluster state)
● Streaming Machine Learning
○ Learn
○ Score
● Join Streams with Updatable Datasets
● [-] Event-time oriented analytics
● [-] Continuous processing

37. Structured Streaming
+

38. Spark Streaming + Structured Streaming
38
val parse: Dataset[String] => Dataset[Record] = ???
val process: Dataset[Record] => Dataset[Result] = ???
val serialize: Dataset[Result] => Dataset[String] = ???
val kafkaStream = spark.readStream…
val f = parse andThen process andThen serialize
val result = f(kafkaStream)
result.writeStream
.format("kafka")
.option("kafka.bootstrap.servers",bootstrapServers)
.option("topic", writeTopic)
.option("checkpointLocation", checkpointLocation)
.start()
val dstream = KafkaUtils.createDirectStream(...)
dstream.map{rdd =>
val ds = sparkSession.createDataset(rdd)
val f = parse andThen process andThen serialize
val result = f(ds)
result.write.format("kafka")
.option("kafka.bootstrap.servers", bootstrapServers)
.option("topic", writeTopic)
.option("checkpointLocation", checkpointLocation)
.save()
}
Structured StreamingSpark Streaming

39. Streaming Pipelines (example)
Structured Streaming
Keyword
Extraction
Keyword
Relevance
Similarity
DB Storage

40. Structured Streaming
New Project?
80%
20%

41. lightbend.com/fast-data-platform

42. Features
1. One-click component installations
2. Automatic dependency checks
3. One-click access to install logs
4. Real-time cluster visualization
5. Access to consolidated production logs
Benefits:
1. Easy to get started
2. Ready access to all components
3. Increased developer velocity
Fast Data Platform Manager, for Managing Running Clusters

43. lightbend.com/learn

44. If you’re serious about having end-to-end monitoring for
your Fast Data and streaming applications,
let’s chat!
SET UP A 20-MIN DEMO