This document discusses the history and future of big data and distributed computing frameworks. It notes that the amount of data in the world has grown enormously from 130GB in 1975 to 2.9ZB in 2012. Hadoop is an open-source software framework that supports distributed applications on large datasets using a distributed file system (HDFS) and MapReduce programming model. Main contributors to the ecosystem include HDFS, MapReduce, HBase, Hive, Pig, ZooKeeper and projects in the Apache incubator like Flume. The future includes improving high availability, scalability, and alternative processing models in core Hadoop and further developing the ecosystem through projects like Apache BigTop.

1. THE ANSWER
TO THE QUESTION
OF THE DATA
eleks by Victor Haydin
DevTalks #1

2. Gordon Moore

3. 1975 2012
Cost of 1 TB storage
$208 000 000 $110
Cost of 1 GFLOPS/s computing facility
$62 000 000 $1.50
Number of network hosts
57 > 1 000 000 000
World’s data amount
~130 GB ~2.9 ZB

4. 1 ZB = 1 000 000 000 000 000 000 000 B
(1021)

8. Commodity Hardware

11. Wikipedia: “Apache Hadoop is a software
framework that supports data-intensive
distributed applications”

12. Main Contributors

14. HDFS: Hadoop Distributed File System
Hardware Failure
Streaming Data Access
Large Data Sets
Simple Coherency Mode (write-once)
Portability

16. Moving Computation is cheaper then
moving Data

17. MapReduce

18. Map(k1,v1) → list(k2,v2)
void map(string key, string value):
for each word w in value:
yield return KeyValuePair(w, 1);
Reduce(k2, list (v2)) → list(v3)
void reduce(string key, int[] values):
int sum = 0;
for each pc in values:
sum += pc;
return KeyValuePair(key, sum);

21. Demo

22. Ecosystem
ZooKeeper

23. 45K nodes, 180-200 PB
3K+ nodes, 36+ PB

24. powered by

25. Future
Core:
• HDFS: high-availability and scalability
• MapReduce: modularity and alternative ways to perform queries
Ecosystem development:
• Apache BigTop: consolidation project
• HBase, Hive, Pig, ZooKeeper, Avro, Sqoop: stabilizing,
interoperability
• Incubator: Flume, Ozzie, Whirr

26. Demo

27. Q&A