This was one of the talks that I gave at the Strata San Jose conference. I migrated my topic a bit, but here is the original abstract: Application developers and architects today are interested in making their applications as real-time as possible. To make an application respond to events as they happen, developers need a reliable way to move data as it is generated across different systems, one event at a time. In other words, these applications need messaging. Messaging solutions have existed for a long time. However, when compared to legacy systems, newer solutions like Apache Kafka offer higher performance, more scalability, and better integration with the Hadoop ecosystem. Kafka and similar systems are based on drastically different assumptions than legacy systems and have vastly different architectures. But do these benefits outweigh any tradeoffs in functionality? Ted Dunning dives into the architectural details and tradeoffs of both legacy and new messaging solutions to find the ideal messaging system for Hadoop. Topics include: * Queues versus logs * Security issues like authentication, authorization, and encryption * Scalability and performance * Handling applications that span multiple data centers * Multitenancy considerations * APIs, integration points, and more

1. © 2014 MapR Technologies 1© 2014 MapR Technologies

2. © 2014 MapR Technologies 2
Contact Information
Ted Dunning
Chief Applications Architect at MapR Technologies
Committer & PMC for Apache’s Drill, Zookeeper & others
VP of Incubator at Apache Foundation
Email tdunning@apache.org tdunning@maprtech.com
Twitter @ted_dunning
Hashtags today: #stratahadoop #ojai

3. © 2014 MapR Technologies 3
Don’t Miss These
• Just-in-time optimizing a database
– Me! at 4:20 PM, Room 230 C, today
• Why flow instead of state?
– Me! at 5:10 PM, Room 210 D/H, today
• High Frequency Decisioning
– Jack Norris! at 11:00 PM, Room 210 B/F, tomorrow
• Threat detection on streaming data
– Carol Macdonald! at 3:45 PM, Solutions Theater, tomorrow
• Scaling Your Business … Zeta Architecture
– Jim Scott! at 5:10 PM, Room 210 D/H, tomorrow

4. © 2014 MapR Technologies 4
And Also, a Little Fun
Come jam with us
The Big Data Boys and the Real-time Stream Band
5:50 PM, MapR booth, today

5. © 2014 MapR Technologies 5
Goals
• Real-time or near-time
– Includes situations with deadlines
– Also includes situations where delay is simply undesirable
– Even includes situations where delay is just fine
• Micro-services
– Streaming is a convenient idiom for design
– Micro-services … you know we wanted it
– Service isolation is a key requirement

6. © 2014 MapR Technologies 6
Real-time or Near-time?
• The real point is flow versus state (see talk later today)
• One consequence of flow-based computing is real-time and
near-time become relatively easy
• Life may be a bitch, but it doesn’t happen in batches!

7. © 2014 MapR Technologies 8
Agenda
• Background / micro-services
• Global requirements
• Scale

8. © 2014 MapR Technologies 9
A microservice is
loosely coupled
with bounded context

9. © 2014 MapR Technologies 10
How to Couple Services and Break micro-ness
• Shared schemas, relational stores
• Ad hoc communication between services
• Enterprise service busses
• Brittle protocols
• Poor protocol versioning
Don’t do this!

10. © 2014 MapR Technologies 11
How to Decouple Services
• Use self-describing data
• Private databases
• Infrastructural communication between services
• Use modern protocols
• Adopt future-proof protocol practices
• Use shared storage where necessary due to scale

11. © 2014 MapR Technologies 13
What is the Right Structure for Flow Compute?
• Traditional message queues?
– Message queues are classic answer
– Key feature/bug is out-of-order acknowledgement
– Many implementations
– You pay a huge performance hit for persistence
• Kafka-esque Logs?
– Logs are like queues, but with ordering
– Out of order consumption is possible, acknowledgement not so much
– Canonical base implementation is Kafka
– Performance plus persistence

12. © 2014 MapR Technologies 14
Scenarios
Profile Database

13. © 2014 MapR Technologies 15
The task
?
POS 1
location, t, card #
yes/no?
POS 2
location, t, card #
yes/no?

14. © 2014 MapR Technologies 16
Traditional Solution
POS
1..n
Fraud
detector
Last card
use

15. © 2014 MapR Technologies 17
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

16. © 2014 MapR Technologies 18
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

17. © 2014 MapR Technologies 19
How to Get Service Isolation
POS
1..n
Fraud
detector
Last card
use
Updater
card activity

18. © 2014 MapR Technologies 20
New Uses of Data
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity

19. © 2014 MapR Technologies 21
Scaling Through Isolation
POS
1..n
Last card
use
Updater
POS
1..n
Last card
use
Updater
card activity
Fraud
detector
Fraud
detector

20. © 2014 MapR Technologies 22
Lessons
• De-coupling and isolation are key
• Private data stores/tables are important,
– but local storage of private data is a bug
• Propagate events, not table updates

21. © 2014 MapR Technologies 23
Scenarios
IoT Data Aggregation

22. © 2014 MapR Technologies 24
Basic Situation
Each location
has many
pumps
pump data
Multiple
locations

23. © 2014 MapR Technologies 25
What Does a Pump Look Like
inlet
out let
m ot or
Temperature
Pressure
Flow
Temperature
Pressure
Flow
Winding temperature
Voltage
Current

24. © 2014 MapR Technologies 26
Basic Situation
Each location
has many
pumps
pump data
Multiple
locations

25. © 2014 MapR Technologies 27
pump data
pump data
pump data
pump data
Basic Architecture Reflects Business Structure

26. © 2014 MapR Technologies 28
Lessons
• Data architecture should reflect business structure
• Even very modest designs involve multiple data centers
• Schemas cannot be frozen in the real world
• Security must follow data ownership

27. © 2014 MapR Technologies 29
Scenarios
Global Data Recovery

28. © 2014 MapR Technologies 30
Tokyo
Corporate
HQ

29. © 2014 MapR Technologies 31
Singapore
Tokyo
Corporate
HQ

30. © 2014 MapR Technologies 32
Singapore
Tokyo
Corporate
HQ

31. © 2014 MapR Technologies 33
Singapore
Tokyo
Corporate
HQ

32. © 2014 MapR Technologies 34
Lessons
• Arbitrary number of topics important for simplicity + performance
• Updates happen in many places
• Mobility implies change in replication patterns
• Multi-master updates simplify design massively

33. © 2014 MapR Technologies 35
Converged Requirements

34. © 2014 MapR Technologies 36
What Have We Learned?
• Need persistence and performance
– Possibly for years and to 100’s of millions t/s
• Must have convergence
– Need files, tables AND streams
– Need volumes, snapshots, mirrors, permissions and …
• Must have platform security
– Cannot depend on perimeter
– Must follow business structure
• Must have global scale and scope
– Millions of topics for natural designs
– Multi-master replication and update

35. © 2014 MapR Technologies 37
The Importance of Common API’s
• Commonality and interoperability are critical
– Compare Hadoop eco-system and the noSQL world
• Table stakes
– Persistence
– Performance
– Polymorphism
• Major trend so far is to adopt Kafka API
– 0.9 API and beyond remove major abstraction leaks
– Kafka API supported by all major Hadoop vendors

36. © 2014 MapR Technologies 38
What we do

37. © 2014 MapR Technologies 39
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Over decades of progress,
Unix-based systems have set the
standard for compatibility and
functionality

38. © 2014 MapR Technologies 40
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Hadoop achieves much higher
scalability by trading away
essentially all of this compatibility
Evolution of Data Storage

39. © 2014 MapR Technologies 41
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
MapR enhanced Apache Hadoop by
restoring the compatibility while
increasing scalability and performance
Functionality
Compatibility
Scalability
POSIX

40. © 2014 MapR Technologies 42
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Evolution of Data Storage
Adding tables and streams enhances
the functionality of the base file
system

41. © 2014 MapR Technologies 43
http://bit.ly/fastest-big-data

42. © 2014 MapR Technologies 44
How we do this with MapR
• MapR Streams is a C++ reimplementation of Kafka API
– Advantages in predictability, performance, scale
– Common security and permissions with entire MapR converged data
platform
• Semantic extensions
– A cluster contains volumes, files, tables … and now streams
– Streams contain topics
– Can have default stream or can name stream by path name
• Core MapR capabilities preserved
– Consistent snapshots, mirrors, multi-master replication

43. © 2014 MapR Technologies 45
MapR core Innovations
• Volumes
– Distributed management
– Data placement
• Read/write random access file system
– Allows distributed meta-data
– Improved scaling
– Enables NFS access
• Application-level NIC bonding
• Transactionally correct snapshots and mirrors

44. © 2014 MapR Technologies 46
MapR's Containers
 Each container contains
 Directories & files
 Data blocks
 Replicated on servers
 No need to manage
directly
Files/directories are sharded into blocks, which
are placed into containers on disks
Containers are 16-
32 GB segments of
disk, placed on
nodes

45. © 2014 MapR Technologies 47
MapR's Containers
 Each container has a
replication chain
 Updates are transactional
 Failures are handled by
rearranging replication

46. © 2014 MapR Technologies 48
Container locations and replication
CLDB
N1, N2
N3, N2
N1, N2
N1, N3
N3, N2
N1
N2
N3Container location database
(CLDB) keeps track of nodes
hosting each container and
replication chain order

47. © 2014 MapR Technologies 49
MapR Scaling
Containers represent 16 - 32GB of data
 Each can hold up to 1 Billion files and directories
 100M containers = ~ 2 Exabytes (a very large cluster)
250 bytes DRAM to cache a container
 25GB to cache all containers for 2EB cluster
 But not necessary, can page to disk
 Typical large 10PB cluster needs 2GB
Container-reports are 100x - 1000x < HDFS block-reports
 Serve 100x more data-nodes
 Increase container size to 64G to serve 4EB cluster
 Map/reduce not affected

48. © 2014 MapR Technologies 50
But Wait, There’s More
• Directories and files are implemented in terms of B-trees
– Key is offset, value is data blob
– Internal transactional semantics guarantees safety and consistency
– Layout algorithms give very high layout linearization
• Tables are implemented in terms of B-trees
– Twisted B-tree implementation allows virtues of log-structured merge
tree without the compaction delays
– Tablet splitting without pausing, integration with file system transactions
• Common security and permissions scheme

49. © 2014 MapR Technologies 51
And More …
• Streams are implemented in terms of B-trees as well
– Topics and consumer offsets are kept in stream, not ZK
– Similar splitting technology as MapR DB tables
– Consistent permissions, security, data replication
• Standard Kafka 0.9 API
• Plans to add OJAI for high-level structuring
• Performance is very high

50. © 2014 MapR Technologies 52
Example
Files
Table
Streams
Directories
Cluster
Volume mount point

51. © 2014 MapR Technologies 53
Cluster
Volume mount point

52. © 2014 MapR Technologies 54
Lessons
• API’s matter more than implementations
• There is plenty of room to innovate ahead of the community
• Posix, HDFS, HBASE all define useful API’s
• Kafka 0.9+ does the same

53. © 2014 MapR Technologies 55
Call to action:
Support the Kafka API’s

54. © 2014 MapR Technologies 56
Call to action:
Support the Kafka API’s
And come by the MapR booth
to check out MapR Streams

55. © 2014 MapR Technologies 57

56. © 2014 MapR Technologies 58
Short Books by Ted Dunning & Ellen Friedman
• Published by O’Reilly in 2014 - 2016
• For sale from Amazon or O’Reilly
• Free e-books currently available courtesy of MapR
http://bit.ly/ebook-real-
world-hadoop
http://bit.ly/mapr-tsdb-
ebook
http://bit.ly/ebook-
anomaly
http://bit.ly/recommend
ation-ebook

57. © 2014 MapR Technologies 59
Streaming Architecture
by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)
Free copies at book
signing today
http://bit.ly/mapr-ebook-streams

58. © 2014 MapR Technologies 60
Thank You!

59. © 2014 MapR Technologies 61
Q&A
@mapr maprtech
tdunning@maprtech.com
Engage with us!
MapR
maprtech
mapr-technologies