This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications. In this talk, you learn more about: 1. What is Apache Flink Stack? 2. Batch vs. Streaming Analytics 3. Key Differentiators of Apache Flink for Streaming Analytics 4. Real-World Use Cases with Flink for Streaming Analytics 5. Who is using Flink? 6. Where do you go from here?

1. Apache Flink:
Real-World Use Cases for Streaming
Analytics
Slim Baltagi
@SlimBaltagi
Brazil - Sao Paulo Apache Flink Meetup
March 17th, 2016

2. Agenda
I. What is Apache Flink Stack?
II. Movement from Batch Analytics to
Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
2

3. I. What is Apache Flink stack?
Gelly
Table
HadoopM/R
SAMOA
DataSet (Java/Scala/Python)
Batch Processing
DataStream (Java/Scala)
Stream Processing
FlinkML
Local
Single JVM
Embedded
Docker
Cluster
Standalone
YARN,
Mesos (WIP)
Cloud
Google’s GCE
Amazon’s EC2
IBM Docker Cloud, …
ApacheBeam
ApacheBeam
MRQL
Table
Cascading
Runtime : Distributed
Streaming Dataflow
Zeppelin
DEPLOYSYSTEMAPIs&LIBRARIESSTORAGE
Files
Local
HDFS
S3, Azure Storage
Tachyon
Databases
MongoDB
HBase
SQL
…
Streams
Flume
kafka
RabbitMQ
…
Batch Optimizer Stream Builder
Storm
FlinkCEP
Gelly-Stream
3

4. I. What is Apache Flink stack?
See First Apache Flink meetup in South America that I
gave as a webinar on February 24th 2016. It is titled:
Introduction to Apache Flink: What, How, Why, Who,
Where? https://www.youtube.com/watch?v=YAKdD1rHCxs (Part 1)
See similar talk on February 2nd 2016 that I previously
gave a at the New York City Apache Flink which. Now,
the world’s largest Flink meetup
• Slideshttp://www.slideshare.net/sbaltagi/apacheflinkwhathowwhywhowhe
rebyslimbaltagi-57825047
• Video recording
https://www.youtube.com/watch?v=G77m6Ou_kFA
 Flink Knowledge Base: all resources related to Flink
http://sparkbigdata.com/component/tags/tag/27-flink
4

5. Agenda
I. What is Apache Flink Stack?
II. Movement from Batch Analytics to
Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
5

6. II. Movement from Batch Analytics to Streaming
Analytics
Batch Streaming
High-latency apps Low-latency apps
Static Files Event Streams
Process-after-store Sense-and-respond
Batch processors Stream processors
6

7. What is batch processing?
Many big data sources represent series of events that
are continuously produced. Example: tweets, web logs,
user transactions, system logs, sensor networks, …
Batch processing: These events are collected together
based on the number of records or a certain period of
time (a day for example) and stored somewhere to be
processed as a finite data set.
What’s the problem with ‘process-after-store’ model:
• Unnecessary latencies between data generation and
analysis & actions on the data.
• Implicit assumption that the data is complete after a
given period of time and can be used to make accurate
predictions for example.
7

8. What is stream processing?
 Most data is available as series of events (click
streams, mobile apps data, .. ) continuously
produced by a variety of applications and systems
in the enterprise.
 Data sources are not anymore typical enterprise
sources but new ones such as social media data,
sensor data …
 Data from disparate systems (internally and
externally) can be integrated in a central hub and:
 Made available as low-latency data streams
required for real-time stream processing.
 Loaded into your data warehouse for offline
analysis.
8

9. Factors behind the movement from Batch
Analytics to Streaming Analytics
There is a movement in Big Data processing from Batch
Analytics to Streaming Analytics driven by many factors:
• Data streams: Sensors networks, mobile apps data, ..
• Technology: Rapidly growing open source streaming
analytics tools, vendors innovating in this space, more
mobile devices than human beings, cloud services for
real-time stream processing…
• Business: Organizations are more and more embracing
streaming analytics for faster time to insight and
competitive advantages.
• Customers: Costumers are becoming more and more
demanding for instant responses in the way they are
used to in social networks: twitter, facebook, linkedin… 9

10. Agenda
I. What is Apache Flink Stack?
II. Batch vs. Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
10

11. III. Key Differentiators of Apache Flink for
Streaming Analytics
The 8 Requirements of Real-Time Stream Processing,
Stonebraker et al. 2005
• Original paper http://cs.brown.edu/~ugur/8rulesSigRec.pdf
• A short summaryhttp://blog.acolyer.org/2014/12/03/the-8-requirements-of-
real-time-stream-processing/
Apache Flink fulfills all these requirements and more!
• http://data-artisans.com/real-time-stream-processing-the-next-step-for-apache-flink/
• http://data-artisans.com/flink-0-10-a-significant-step-forward-in-open-source-stream-
processing/
• http://data-artisans.com/flink-1-0-0/
• https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison
• https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
778DAD7eqw5GANwE/edit
• http://www.slideshare.net/robertmetzger1/january-2016-flink-community-update-roadmap-
2016/9
11

12. III. Key Differentiators of Apache Flink for
Streaming Analytics
True Low latency streaming engine: fast results in milliseconds
High throughput: handle large data amounts (millions of events per second)
• http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
Exactly once guarantees: Correct results, also in failure cases
• http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-
processing-with-apache-flink/
Programmability: Higher level, Intuitive and easy to use APIs
Backpressure refers to the situation where a system is receiving data at a
higher rate than it can process during a temporary load spike.
• http://data-artisans.com/how-flink-handles-backpressure/
Event time and out of order stream processing
• http://data-artisans.com/how-apache-flink-enables-new-streaming-applications-
part-1/
Stateful stream processing and versioning state
• http://data-artisans.com/how-apache-flink-enables-new-streaming-applications/ 12

13. Agenda
I. What is Apache Flink Stack?
II. Batch vs. Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
13

14. IV. Real-World Use Cases with Flink for Streaming
Analytics
Stonebraker et al. make the case in 2005 that stream
processing is going to become increasingly important.
Not just for the usual finance, fraud, and command-and-
control use cases, but also….… “as the “sea change”
caused by cheap micro-sensor technology takes hold, we expect
to see everything of material significance on the planet get
“sensor-tagged” and report its state or location in real time. This
sensorization of the real world will lead to a “green field” of novel
monitoring and control applications with high-volume and low-
latency processing requirements.”
Reference:http://blog.acolyer.org/2014/12/03/the-8-requirements-of-real-time-
stream-processing/
14

15. Shift from Reactive approach to proactive
approach
Capturing new data and providing the ability to
process streams of this data is allowing
organizations to shift
• From: taking a REACTIVE, post transaction
approach
• To: more of a PROACTIVE, pre decision approach
to interactions with their customers, suppliers and
employees.
Again, no matter the vertical, this transition is
happening.
15

16. …to real-time
personalization
From static
branding
…to repair before
break
From break then
fix
…to designer
medicine
From mass
treatment
…to automated
algorithms
From educated
investing
…to 1x1 targeting
From mass
branding
A shift in Advertising
A shift in Financial Services
A shift in Healthcare
A shift in Retail
A shift in Manufacturing
Big Data Analytics
Frameworks enable
shifting the business
from…
Reactive
Proactive
Shift from Reactive approach to proactive approach
16

17. Real-Time Monitoring of Customer Activity
Events
17

18. Generic Streaming Analytics Architectural pattern.Event
Producers
EventCollector
EventBroker
EventProcessor
Indexer
Visualizer/Search
• Kafka
• RabitMQ
• JMS
• Flink
• Spark
• Storm
• Samza
• ElasticSearch
• Solr
• Cassandra
• NoSQL DB
• Kibana
• Custom
GUI
• Flume
• SpringXD
• Logstash
• Nifi
• Fluentd
• Apps
• Devices
• Sensors
18

19. IV. Real-World Use Cases with Flink for Streaming
Analytics
Below is list several use cases, taken from real
industrial situations:
Financial Services
– Real-time fraud detection.
– Real-time mobile notifications.
Healthcare
– Smart hospitals - collect data and readings from hospital
devices (vitals, IVs, MRI, etc.) and analyze and alert in real
time.
– Biometrics - collect and analyze data from patient devices
that collect vitals while outside of care facilities.
Ad Tech
– Real-time user targeting based on segment and preferences.
Oil & Gas
• Real-time monitoring of pumps/rigs. 19

20. IV. Real-World Use Cases with Flink for
Streaming Analytics
Retail
• Build an intelligent supply chain by placing sensors or RFID
tags on items to alert if items aren’t in the right place, or
proactively order more if supply is low.
• Smart logistics with real-time end-to-end tracking of delivery
trucks.
Telecommunications
• Real-time antenna optimization based on user location data.
• Real-time charging and billing based on customer usage,
ability to populate up-to-date usage dashboards for users.
• Mobile offers.
• Optimized advertising for video/audio content based on what
users are consuming.
20

21. Agenda
I. What is Apache Flink Stack?
II. Batch vs. Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
21

22. V. Who is using Flink?
ho is using Apache Flink?How companies are using Flink as presented at Flink
Forward 2015. Kostas Tzoumas and Stephan Ewen.
http://www.slideshare.net/stephanewen1/flink-use-cases-bay-area-meetup-
october-2015
Powered by Flink page:
 https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
22

23. V. Who is using Flink? is using Apache
Flink? has its hack week and the winner was
a Flink based streaming project! December 18, 2015
• Extending the Yahoo! Streaming Benchmark and Winning
Twitter Hack-Week with Apache Flink. Posted on February
2, 2016 by Jamie Grier http://data-artisans.com/extending-the-
yahoo-streaming-benchmark/
 did some benchmarks to
compare performance of their use case implemented
on Apache Storm against Spark Streaming and Flink.
Results posted on December 18, 2015
• http://yahooeng.tumblr.com/post/135321837876/benchmarking-
streaming-computation-engines-at
• http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
• https://github.com/dataArtisans/yahoo-streaming-benchmark 23

24. Agenda
I. What is Apache Flink Stack?
II. Batch vs. Streaming Analytics
III. Key Differentiators of Apache Flink for
Streaming Analytics
IV. Real-World Use Cases with Flink for
Streaming Analytics
V. Who is using Flink?
VI. Where do you go from here?
24

25. VI. Where do you go from here?
 A few resources for you:
• Flink at the Apache Software Foundation: flink.apache.org/
• Free ebook from MapR: Streaming Architecture: New
Designs Using Apache Kafka and MapR Streams
https://www.mapr.com/streaming-architecture-using-apache-kafka-mapr-
streams
• Free Apache Flink training from data Artisans
http://dataartisans.github.io/flink-training/ Still version 0.10.1 and not
latest 1.0
• Flink Knowledge Base: One-Stop for everything related to
Apache Flink http://sparkbigdata.com/component/tags/tag/27-flink
• Apache Flink in Action is probably the First book on
Apache Flink! It will be published by Manning. I am co-
authoring this book! Please stay tuned for the MEAP: Manning
Early Access Program!
25

26. VI. Where do you go from here?
 A few takeaways :
• Organizations are more and more embracing streaming
analytics for:
• Use cases requiring lower latency: monitoring,
altering, …
• Faster time to insight
• Competitive advantages
• By leveraging streaming analytics, new startups
are challenging established companies. Example:
Pay-As-You-Go insurance or Usage-Based Auto
Insurance
• Speed is said to have become the new currency of
business.
26

27. Thanks!
To all of you for attending!
Let’s keep in touch!
• sbaltagi@gmail.com
• @SlimBaltagi
• https://www.linkedin.com/in/slimbaltagi
Any questions?
27