1. @just_do_neet
1

2. Hadoop Conference Japan 2011 fall
• 2011/9/26( )
http://hadoop-conference-japan-2011-fall.eventbrite.com/
• Hadoop
1000
•
Cloudera HortonWorks MapR
•
NTT mixi Yahoo
2

3. http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud
3

4. 4

5. 5

6. Hadoop T
6

7. Hadoop
7

8. Hadoop
• Apache Software Foundation OSS
•
•
:MapReduce
•
:HDFS/HBase
8

9. Hadoop
• Google
GFS(ACM ’03) “The Google File System” → HDFS
MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce
BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase
• Yahoo! Inc. Cloudera Doug Cutting Google
2004 Java OSS
”Hadoop” Doug
9

10. Hadoop
•
index
index etc...
※Google
※ Doug Lucene Nutch OSS
• Hadoop
• Hadoop
10

11. Hadoop
• HBase
BigTable
• ZooKeeper
Chubby
• Hive
HDFS SQL
• Flume
• and more....
11

12. • sorry, confidential...
12

13. 13

14. Hadoop Conference Japan 2011 Fall
• Hadoop 0.23
•
•
14

15. Hadoop 0.23
• 2011 Q4 beta
Integration Test Pig
• Hadoop
NameNode SPoF (NameNode
• MapReduce v2 : MapReduce
MPI(C Giraph Hama Graph Spark etc
•
MapReduce shuffle 30% / 10,000 4000 / etc
15

16. Giraph
• Hadoop
https://github.com/aching/Giraph
Graph → Google PageRank
Google Pregel(SIGMOD ’10)
• BSP(Bulk Synchronous Parallel
16

17. Spark
• Scala
http://www.spark-project.org/
→
ex.
• Mesos
http://www.mesosproject.org/
Hadoop
17

18. H
• Cloudera
http://www.cloudera.com/
OSS (CDH Sqoop Hue
• HortonWorks
http://www.hortonworks.com/
Yahoo! Inc. Hadoop
• MapR
http://www.mapr.com/
Hadoop
•
Hadoop
18

19. H Cloudera
• CDH
Linux
• OSS
Sqoop SQL HDFS Hue Web Console etc..
• SCM Express
Web Hadoop
50 free
• Cloudera Enterprise
Cloudera Management Suite
•
19

20. H HortonWorks
• Yahoo! Inc. Hadoop 22
Hadoop ※ (
500,000
• 2011/7
• Yahoo! Inc.
Yahoo! Mail
42,000 Hadoop
• Horton the Elephant
http://en.wikipedia.org/wiki/Horton_the_Elephant
20

21. H MapR
• Hadoop
EMC OEM Greenplum
http://www.greenplum.com/
•
HDFS C++ HDFS
random read/write
NIC bonding
snapshot etc..
NFS Hadoop
R
• OSS Hadoop
21

22. • Hadoop
LSP etc..
• Asakusa Framework OSS
https://github.com/asakusafw
DSL
22

23. Yahoo! Japan
•
Hadoop
Geohash Hadoop Iterable
Twitter
※ MPI ...
23

24. 24

25. • Hadoop
BtoC
Hadoop 0.23 MapReduce v2 Hadoop
• Hadoop Hadoop
Map/Reduce + Graph
BI
25

26. 26