http://flink-forward.org/kb_sessions/faster-and-furiouser-flink-drift/ Not long ago, we had the opportunity to test Apache Flink to see just how fast it would go on a moderately realistic task with fast hardware and with a good streaming transport layer underneath. Our goal was not so much careful comparison with other software, but flat-out speed, Flink against Flink. In the process, we learned a lot about what it takes to go fast. Some of the lessons were ones that we had “learned” a number of times before: – the bottleneck isn’t where you thought it was – copying data is expensive – context switches are expensive – measure twice, cut once But there were some real surprises along the way. The really important knobs weren’t quite what people say you should turn. One of the biggest surprises was the degree to which high performance libraries have threading built into them which makes the actual concurrrency much higher than the apparent concurrency. The result was that at least one cluster parameter needed to be adjusted by 30x to get real

1. © 2014 MapR Technologies 1© 2014 MapR Technologies

2. © 2014 MapR Technologies 2
Me, Us
• Ted Dunning, MapR Chief Application Architect, Apache Member
– Committer PMC member Zookeeper, Drill, others
– Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin
– VP Incubator
– Bought the beer at the first HUG
• MapR
– Produces first converged platform for big and fast data
– Includes data platform (files, streams, tables) + open source
– Adds major technology for performance, HA, industry standard API’s
• Contact
@ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com

3. © 2014 MapR Technologies 3
New book on Apache Flink
Download free pdf
courtesy of MapR Technologies
mapr.com/flink-book

4. © 2014 MapR Technologies 4
Agenda
• Why streaming first architecture
• What does fast mean?
• How do I make something fast?
• Minor pause for reality check
• First steps … heavy bottlenecks
• Real results
• Deeper insights

5. © 2014 MapR Technologies 5
Is this really a
revolutionary moment?

6. © 2014 MapR Technologies 6
Scenario:
Profile Database

7. © 2014 MapR Technologies 7
The task
?
POS 1
location, t, card #
yes/no?
POS 2
location, t, card #
yes/no?

8. © 2014 MapR Technologies 8
Traditional Solution
POS
1..n
Fraud
detector
Last card
use

9. © 2014 MapR Technologies 9
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

10. © 2014 MapR Technologies 10
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

11. © 2014 MapR Technologies 11
How to Get Service Isolation
POS
1..n
Fraud
detector
Last card
use
Updater
card activity

12. © 2014 MapR Technologies 12
New Uses of Data
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity

13. © 2014 MapR Technologies 13
Scaling Through Isolation
POS
1..n
Last card
use
Updater
POS
1..n
Last card
use
Updater
card activity
Fraud
detector
Fraud
detector

14. © 2014 MapR Technologies 14
For this to work (socially),
streaming has to be faster
than almost any requirement

15. © 2014 MapR Technologies 15
So how do we make something
go really fast?

16. © 2014 MapR Technologies 16
Make some
data
Process itmove it

17. © 2014 MapR Technologies 17
Make some
data
Process itmove it
World
domination
move it

18. © 2014 MapR Technologies 18
Well, perhaps not quite so
simple?

19. © 2014 MapR Technologies 19
Interactive recommendation
query
db
Off-line analysis
Real-time event
source
Recent
history
Item
linkage
Search Recommendations
Cooccurrence
analysis
Long-
term
history
queue
queue
Recommendations

20. © 2014 MapR Technologies 20
mySQL
mySQL
files
Web-site
Auth
service
Upload
service
Image
extractor
Transcoder
User
profiles
Search
User action
logging
Recommendation
analysis
mySQL
mySQL
mySQL
Oracle
Solr
Elastic
User Generated Content

21. © 2014 MapR Technologies 21
Yahoo Streaming Benchmark
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time

22. © 2014 MapR Technologies 22
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Client lock Partitions Threads/machineShuffle

23. © 2014 MapR Technologies 23
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Client lock Partitions Threads/machineShuffle
Threads/machineShuffle

24. © 2014 MapR Technologies 24
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Ad server impressions Filter Project
Augment
Group by
campaign
Count
impressions
Client lock Partitions Threads/machineShuffle
Threads/machineShuffle

25. © 2014 MapR Technologies 25
What we do at MapR

26. © 2014 MapR Technologies 26
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Over decades of progress,
Unix-based systems have set the
standard for compatibility and
functionality

27. © 2014 MapR Technologies 27
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Hadoop achieves much higher
scalability by trading away
essentially all of this compatibility
Evolution of Data Storage

28. © 2014 MapR Technologies 28
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
MapR enhanced Apache Hadoop by
restoring the compatibility while
increasing scalability and performance
Functionality
Compatibility
Scalability
POSIX

29. © 2014 MapR Technologies 29
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Evolution of Data Storage
Adding converged tables and streams
enhances the functionality of the base
file system

30. © 2014 MapR Technologies 30
http://bit.ly/fastest-big-data

31. © 2014 MapR Technologies 31
Key Ideas
• Convergence of files, tables, streams into single platform
– All forms of persistence share common implementation base
• Very high abstraction from hardware … no need to provision
clusters for tables and files
– Common disaster recovery, security, availability models for files,
directories, tables and streams
• Very high performance levels

32. © 2014 MapR Technologies 32
Key Issues
• MapR itself is heavily threaded internally (as many as 50k
threads/core)
• MapR client can have multiple internal threads
• Ordering boundaries require serialization, locks or memory
contention
– At client level and also within single stream/topic/partition
• Replication, splitting, data location completely automated by
default, explicit control available
• MapR Streams and Flink are in same cluster, but some shuffles
still required

33. © 2014 MapR Technologies 33
Initial Configuration
• 10 nodes in cluster
• 1 Flink task manager / node
• 72 partitions in impressions stream
• Each task manager spawns 72
generator threads
Ad server impressions
Ad server impressions
10x72 threads
72 partitions
• At full speed, partition insert points wander around cluster to
avoid hot-spotting
• MapR client connection shared by all threads in task
manager. Having more client connections could help

34. © 2014 MapR Technologies 34
Tuning #1
• Large number of threads and single client connection per node
caused massive contention at serialization point inside client
• Switched to 3 Flink task managers per node
• 2 task managers each run 1 producer thread
– More data pushed by 1 thread than previously sent by 72

35. © 2014 MapR Technologies 35
Tuning #2
• Effective cluster-wide parallelism limited by 72 partitions in
stream
• Increasing to 300 partitions substantially improved performance

36. © 2014 MapR Technologies 36
The consumer
• Initial tuning had 72 consumer threads per
node
• Final tuning used single consumer thread
per Flink task manager
Filter
Campaign
info
Project
Augment
Filter Project
Augment

37. © 2014 MapR Technologies 37
The Shuffle / Group-by
• Shuffles were also run by the
single consumer task
manager
• Even with shuffle, consumer
processes balanced
producer processes
Group by
campaign
Count
impressions Results
Window by event time
Group by
campaign
Count
impressions

38. © 2014 MapR Technologies 38
Tuning #3
• In separate experiments, number of campaigns was increased to
1e6 from original 100
• This caused bottle neck to shift massively to data export step
• Serving results directly from Flink memory avoids this step

39. © 2014 MapR Technologies 39
Final Comparisons
Flink on MapR
no tuning
Transactions / second
(millions)
0 5 10 15
Flink on MapR
tuned
Final result for tuning was
250% improvement
No serious optimization was
required, however

40. © 2014 MapR Technologies 40
The Moral
• Default of 10 partitions per topic is fine for large-scale multi-
tenancy, but special purpose applications may need tuning to
higher levels (we ended up with 30 partitions per node)
• Asynchronous client gives effective threading with small number
of producer threads, large number of producer threads was
counter-productive
• Net speedup of 250% with tuning, so far
• Gut feel is that there is ~4x more performance still to come

41. © 2014 MapR Technologies 41
Me, Us
• Ted Dunning, MapR Chief Application Architect, Apache Member
– Committer PMC member Zookeeper, Drill, others
– Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin
– VP Incubator
– Bought the beer at the first HUG
• MapR (www.mapr.com)
– Produces first converged platform for big and fast data
– Includes data platform (files, streams, tables) + open source
– Adds major technology for performance, HA, industry standard API’s
• Contact
@ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com

42. © 2014 MapR Technologies 42
New book on Apache Flink
Download free pdf
courtesy of MapR Technologies
mapr.com/flink-book

43. © 2014 MapR Technologies 43
Streaming Architecture
by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)
Free signed hard copies at
MapR booth at Flink
Forward
http://bit.ly/mapr-ebook-streams

44. © 2014 MapR Technologies 44
Short Books by Ted Dunning & Ellen Friedman
• Published by O’Reilly in 2014 - 2016
• For sale from Amazon or O’Reilly
• Free e-books currently available courtesy of MapR
Download pdfs: mapr.com/ebooks-pdf

45. © 2014 MapR Technologies 45
Thank You!

46. © 2014 MapR Technologies 46
Q&A
@mapr maprtech
tdunning@maprtech.com
Engage with us!
MapR
maprtech
mapr-technologies