Flink started with the mission to unify batch and stream processing. We believe that Flink’s architecture is uniquely positioned to be a great engine for streaming, batch and AI workloads at the same time. We will talk about the work we did in this direction.

1. Unified Engine for Data Processing and AI
Xiaowei Jiang
Sept, 2018

2. Stratify Inc.
2000 - 2002
Member of Technical Staff
2002
Microsoft
2002 - 2010
Principal Software Engineer
Facebook
2010 - 2014
Software Engineer
Alibaba Group
2014 - Now
Senior Director
2010 2014 Now
About Me

3. EB Total PB Everyday 472M Events/sec1T Event/Day
About Alibaba

4. 11.11 DashBoard
472M events/s Exactly-onceSub-second
latency
Highly Available

5. Low Latency
Fixed Query
Periodic/Continuous
Batch Jobs
High Latency
Flexible Query
Stream Processing Progressive Processing Batch Analytics
Streaming as the core abstraction, Batch as a special case of Streaming
Scenarios for Data Processing

6. Runtime
Table API & SQL
Relational
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming DataFlow
Local
Single JVM
Cluster
Standalone/YARN
Cloud
GCE/EC2
DataStream API
Stream Processing
DataSet API
Batch Processing
Transformation
StreamGraph
Operator Tree
Batch Plan
Optimized Plan
Job Graph
Stream Task & OP Batch Task & Driver
Very different code for
Stream & Batch
Different API for Stream
and Batch Processing
Flink Architecture
API

7. Apache Flink is the
most sophisticated
open-source Stream Processor
Stream Processing Engines
Stream Processing
samza Apex

8. Can Apache Flink become the
most sophisticated
open-source batch processor?
Batch Processing
Batch Processing Engines

9. Batch Micro-benchmark
1000
1120
1240
1360
1480
1600
1720
1840
1960
2080
2200
Hadoop-2.7.1 Tez-0.7.0 Spark-1.5.1 Flink-0.9.1
1,480
2,171
1,887
2,157
Flink is the fastest due to its pipelined execution
Tez and Spark do not overlap 1st and 2nd stages
MapReduce is slow despite overlapping stages
Result of sorting 80GB/node (3.2TB)
A Comparative Performance Evaluation of Flink, Dongwon Kim, POSTECH, Flink Forward 2015

10. Functionality
Unified Engine
Performance
Reliability

11. Declarative
Optimizable
Understandable
Stable
Unified
Why SQL

12. Stream Data
Flink
Batch Data
Dynamic
Table
Continuous
Query
Dynamic
Table
Stream Data
Batch Data
Stream Job
Batch Job
Dynamic Table

13. Streaming SQL Functionalities
Stream Join Agg w/ Retraction Window
UDX DDL Support Connector

14. Batch SQL Completion
TPCH & TPCDS 2K+ CASE & SQL

15. SrcA SrcB
UnionAll
Filter
Probe Build
HashJoin
Filter
Stream Model Batch Model
Existing Execution Model

16. Unified Operator Framework
Unified Operator Abstraction
• Operators can choose inputs
• Operators can be chained easily
• Helps batch as well as streaming
Unified
Operator

17. Table API & SQL
Relational
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming DataFlow
Local
Single JVM
Cluster
Standalone/YARN
Cloud
GCE/EC2
New DesignOld Design
Flink Architecture: New Design
DataSet
API
Runtime
DDAG API & Stream Operators
Local
Single JVM
Cluster
Standalone/YARN
Cloud
ECS/EC2
Query Processor
Query Optimization & Query Execution
DataStream
API
SQL/TableAPI
Relational

18. JobGraphPhysical PlanTable API & SQL Logical Plan
Pluggable
Catalog
Planner
SubQuery
Decorrelation
Filter/Project
Push-Down
Join
Reorder
…
Configurable
Rules
Query Processing

19. Functionality
Unified Engine
Performance
Reliability

20. Batch Performance
TPCH Performance (the Lower, the Better)
TPCH
CASE
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Blink Flink 1.6.0

21. TableA
Probe
TableA
Probe
Scheduling
Customizable Scheduling
• Flexible control over when tasks get scheduled
• Much better resource usage achieved
TableB
Build
HashJoin
TableB
Build
HashJoin

22. A B
Only One-Input Operators can be chained
A B
Multi-Input Operators can also be chained
Flexible Chaining

23. 0x000… “awesome” “flink”321 32L 7 39L 5
Fixed length part Variable length part
Record Format
• Introduced new row format: BinaryRow
• Tight integration with memory management
• Avoid deserialization cost
Record Format

24. Performant
Operators
Operator codegen
HashAgg
Improved HashJoin
Semi/Anti join
Resource
Optimizations
Stats based estimation
Dynamic memory allocation
Expression
Optimizations
Operate directly on binary data
JVM intrinsics
Hot method codegen
Query Execution

25. Rich Stats
• NDV
• NULL count
• Avg length
• Max length
• Min
• Max
Query Optimization
Cost Based
• Join order
• Join type
• Agg strategy
• ……
Advanced Rules
• Subplan reuse
• Join condition expansion
• Shuffle removal
• Distinct Agg rewrite
• ….

26. Performance Improvements - Recap
10x
Overall
2
Runtime
2+
Query Execution
2+
Query Optimizer
= x x

27. Functionality
Unified Engine
Performance
Reliability

28. Reliability Improvements
Failover
• Region Based Failover
• JM Failover
• Blacklist
• ….
More Details?
Sept 4th, 2018
5:10 PM - 5:50 PM
Maschinenhaus
Feng Wang, Alibaba
Runtime Improvements for Flink Batch
Processing
Shuffle
• Decoupled from TM
• Yarn Shuffle Service
• Async mode

29. Next Steps
Grand Unification of Data Processing
Flink Machine Learning/AI
Switch between batch processing and streaming
seamlessly
PyFlink, TableAPI, DL Integration, Flink ML Improvements
Next Steps

30. Already the best
stream processor
Becoming the best
batch processor
Unified approach
benefits both batch &
stream processing
Working on seamless
experience for AI
Takeaway

31. Flink Forward China
Flink Forward China
Dec 20th-21st @ Beijing National Conference Center
First Flink Forward Conference in Asia，3000+ participants expected
Flink Community
Joint efforts by all major players in Flink community from China
Call For Talks & Sponsors
Submit your talks to flink-forward-china@list.alibaba-inc.com