Hadoop Map Reduce

Hadoop Map Reduce Apurva Jadhav Senior Software Engineer TheFind Inc. (diagrams borrowed from various sources)

Introduction ,[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Core ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Typical cluster ,[object Object],[object Object]

Hadoop Distributed File System ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HDFS Architecture Namenode (Master) Datanode Client Datanode Datanode Read Write Replication Meta data ops Stores FS metadata – namespace, block locations Stores the data blocks as linux files

Hadoop Map Reduce ,[object Object],[object Object],[object Object],[object Object],[object Object]

Word Count Problem ,[object Object],[object Object],[object Object],[object Object],[object Object]

Why Map Reduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Map Reduce Architecture ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Jobtracker tasktracker tasktracker tasktracker Input Job (mapper, reducer, input) Data transfer Assign tasks

Map Reduce Programming Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Map Reduce Programming Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Word Count Map Reduce Job ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Map Reduce Architecture - 2 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Map Reduce Architecture - 2 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Jobtracker tasktracker tasktracker tasktracker Input Job (mapper, reducer, input) Data transfer Assign tasks

Map Reduce Advantages ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Conclusion ,[object Object],[object Object],[object Object],[object Object]

References ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Map Reduce

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hadoop Map Reduce

Similar to Hadoop Map Reduce (20)

More from VNIT-ACM Student Chapter

More from VNIT-ACM Student Chapter (12)

Recently uploaded

Recently uploaded (20)

Hadoop Map Reduce