Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Robust Stream Processing With Apache Flink

519 views

Published on

My talk from Reactive Summit 2016 - Not terribly interesting without the actual talk since most of it was live coding and demo.

Published in: Technology
  • Login to see the comments

Robust Stream Processing With Apache Flink

  1. 1. Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier jamie@data-artisans.com
  2. 2. Who am I? • Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing
  3. 3. Overview • What is Apache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing
  4. 4. What is Apache Flink? s an open source platform for distributed stream and batch da
  5. 5. What is Apache Flink?
  6. 6. Stream Processing Your Code Data Stream Data Stream
  7. 7. Stateful Stream Processing Your Code Data Stream Data Stream State
  8. 8. More Complex Example Rabbit MQ Files Kafka Filter Map Join / Sum Influx DB C*
  9. 9. Distributed and Parallel Deployment MapR Stream s Files Kafka Filter Pars e Join / Sum Influx DB C*
  10. 10. Benchmarking on HPC Cluster 0 18,000,000 36,000,000 54,000,000 72,000,000 90,000,000 Throughput: msgs/sec 10 Machines with 40 GigE 72 Million msgs/sec
  11. 11. Robust Stream Processing with Apache Flink
  12. 12. Code Example!
  13. 13. Amplifier Function Amplifier Amplified Stream State* *State: Amplification factors for each key
  14. 14. Windowing
  15. 15. Processing Time vs Event Time
  16. 16. Windowing in Processing Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time
  17. 17. Windowing in Event Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time
  18. 18. Processing Time = Errors!
  19. 19. Event Time = Accuracy
  20. 20. Failure Handling
  21. 21. Downtime Handling
  22. 22. Data Reprocessing
  23. 23. We’re Hiring! http://data-artisans.com/careers
  24. 24. Questions?
  25. 25. Thanks!

×