it talks about the evolution of big data computation architecture

1. Evolution of Big Data
Architecture
Gang Tao

2. Computing Trend

4. Non Functional Requirements
• Latency
• Throughput
• Fault Tolerant
• Scalable
• Exactly Once Semantic

5. Hadoop

6. Hadoop Eco System

7. Map Reduce
Move computation to Data

8. Map Reduce

9. Hadoop the Limitation
• Map/Reduce is hard to use
• Latency is high
• inevitable data movement

10. Lambda

11. Lambda Architecture
query = function(all data)

12. Design Principle
human fault-tolerance
– the system is unsusceptible to data loss or data corruption because
at scale it could be irreparable.
data immutability
– store data in it’s rawest form immutable and for perpetuity. (INSERT/
SELECT/DELETE but no UPDATE !)
recomputation
– with the two principles above it is always possible to (re)-compute
results by running a function on the raw data.

13. Lambda Architecture

14. Batch Processing

15. Batch Processing
Using only batch processing, leaves you always with a portion
of non- processed data.

16. Realtime Stream Processing

17. query = function(all data) +
function(new data)

18. Lambda Architecture

19. A Reference Case

20. Lambda the Bad?
What is Good?
What is Bad?
Inmutable Data , Reprocessing
Keeping code written in two
different systems perfectly in sync

21. Over complicated Lambda

22. Batch VS. Stream

23. Stream & Realtime

24. Storm Logic View
Bolts
Spouts
Move data to Computation

25. Storm Deployment View

26. Code Compare

27. DAG
• DAG : Directed Acyclic Graph
• Used in Spark, Storm, Flink etc.

28. Out of Control
Build Complexity with Simplicity

29. Stream Processing Model
One at a time Micro batch

30. Stream Processing Model
One at a time Micro Batch
Low Latency Y N
High Throughput N Y
at least once Y Y
excatly once Sometimes Y
simple programing model Y N

31. Stream Computing the Limitation
• Queries must be written before data
• There should be another way to query past data
• Queries cannot be run twice
• All results will be lost when any error occurs All data have gone
when bugs found
• Disorders of events break results
• Recorded time based queries? Or arrival time based queries?

32. Batch/Stream Unification

33. Stream with Spark

34. Apache Flink

35. Flink Distributed Snapshot

36. Fault Tolerance in Stream
• At Least Once : ensure all operators see all events
• Stream -> Replay on failure
• Exactly Once :
• Flink : distributed Snapshot
• Spark : Micro Batch

37. Jay Kreps

38. Kappa Architecture