My talk from Reactive Summit 2016 - Not terribly interesting without the actual talk since most of it was live coding and demo.

1. Robust
Stream Processing
with
Apache Flink
Jamie Grier
@jamiegrier
jamie@data-artisans.com

2. Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

3. Overview
• What is Apache Flink?
• What is Stateful Stream Processing?
• Windowed computation over streams
• Robust Time Handling (Event Time vs Processing Time)
• Robust Failure Handling
• Robust Planned Downtime Handling
• Robust Reprocessing

4. What is
Apache Flink?
s an open source platform for distributed stream and batch da

5. What is
Apache Flink?

6. Stream Processing
Your
Code
Data Stream Data Stream

7. Stateful
Stream Processing
Your
Code
Data Stream Data Stream
State

8. More Complex
Example
Rabbit
MQ
Files
Kafka
Filter
Map
Join /
Sum
Influx
DB
C*

9. Distributed and Parallel
Deployment
MapR
Stream
s
Files
Kafka
Filter
Pars
e
Join /
Sum
Influx
DB
C*

10. Benchmarking on
HPC Cluster
0 18,000,000 36,000,000 54,000,000 72,000,000 90,000,000
Throughput: msgs/sec
10 Machines with 40 GigE
72 Million msgs/sec

11. Robust Stream Processing
with Apache Flink

12. Code Example!

13. Amplifier Function
Amplifier
Amplified Stream
State*
*State: Amplification factors for each key

14. Windowing

15. Processing Time
vs
Event Time

16. Windowing in Processing
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time

17. Windowing in Event
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time

18. Processing Time = Errors!

19. Event Time = Accuracy

20. Failure Handling

21. Downtime Handling

22. Data Reprocessing

23. We’re Hiring!
http://data-artisans.com/careers

24. Questions?

25. Thanks!