Introduction To Map Reduce

Introduction to MapReduce, an Abstraction for Large-Scale Computation Ilan Horn Google, Inc. (most slides borrowed from Jeff Dean)

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Problem: lots of data ,[object Object],[object Object],[object Object],[object Object],[object Object]

Solution: spread the work over many machines ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Computing Clusters ,[object Object],[object Object]

Machines ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Implications of our Computing Environment ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

MapReduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Typical problem solved by MapReduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More specifically… ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Example: Word Frequencies in Web Pages ,[object Object],[object Object],[object Object],[object Object],“ document1”, “to be or not to be” “ to”, “1” “ be”, “1” “ or”, “1” …

Example continued: word frequencies in web pages ,[object Object],[object Object],[object Object],“ be”, “2” “ not”, “1” “ or”, “1” “ to”, “2” key = “or” values = “1” “ 1” key = “be” values = “1”, “1” “ 2” key = “to” values = “1”, “1” “ 2” key = “not” values = “1” “ 1”

Example: Pseudo-code ,[object Object],[object Object],[object Object]

Widely applicable at Google ,[object Object],[object Object],[object Object],web access log stats web link-graph reversal inverted index construction statistical machine translation … distributed grep distributed sort term-vector per host document clustering machine learning ...

Example: Generating Language Model Statistics ,[object Object],[object Object],[object Object],[object Object],[object Object]

Example: Joining with Other Data ,[object Object],[object Object],[object Object],[object Object]

MapReduce Programs in Google’s Source Tree

New MapReduce Programs Per Month Summer intern effect

MapReduce: Scheduling ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Parallel MapReduce Map Map Map Map Input data Reduce Shuffle Reduce Shuffle Reduce Shuffle Partitioned output Master

Task Granularity and Pipelining ,[object Object],[object Object],[object Object],[object Object],[object Object]

Fault tolerance: Handled via re-execution ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Refinement: Backup Tasks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Refinement: Locality Optimization ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Refinement: Skipping Bad Records ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Other Refinements ,[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Performance Results & Experience

Usage Statistics Over Time 157 193 758 3,288 217 634 Aug, ‘04 29 268 2,970 6,743 52,254 2,002 874 Mar, ‘06 172 232 Average worker machines 941 Output data written (TB) 2,756 Intermediate data (TB) 12,571 Input data read (TB) 981 Machine years used 934 Average completion time (secs) Mar, ‘05 72 Number of jobs (1000s) 394 14,018 34,774 403,152 11,081 395 Sep, ‘07 2,217

Implications for Multi-core Processors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Introduction To Map Reduce

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Introduction To Map Reduce

Similar to Introduction To Map Reduce (20)

Recently uploaded

Recently uploaded (20)

Introduction To Map Reduce