SlideShare a Scribd company logo

Hadoopソースコードリーディング MapReduce障害時のフロー

•Download as KEY, PDF•
•
Ryu Kobayashi
Ryu Kobayashi

This document discusses Hadoop and MapReduce. It describes the interactions between the JobTracker, TaskTrackers, and how tasks are assigned and monitored. It also mentions configuration options for Ubuntu/Debian for IPv6 and task failure handling in MapReduce jobs.

Technology
1 of 18
2 Hadoop MapReduce
• • Twitter @ryu_kobayashi • ...
• MapReuceTaskTracker JobTracker
• MapReuceTaskTracker JobTracker • Hadoop P176-178
• MapReuceTaskTracker JobTracker • Hadoop P176-178 •
•
• • Ubuntu Debian
• • Ubuntu Debian • Ubuntu IPv6 OS -Djava.net.preferIPv4Stack=true
• • Ubuntu Debian • Ubuntu IPv6 OS -Djava.net.preferIPv4Stack=true • OS •
MapReduce
• JobClient#runJob • JobClint#monitorAndPrintJob mapred-site.xml jobclient.output.filter FAILDE monitorAndPrintJob Job
TaskTracker • TaskTracker#run • TaskTracker#offerService • TaskTracker#markUnresponsiveTasks Task • JobTracker#statusUpdate • TaskInProgress#reportProgress • JobTracker TaskTracker#offerService HeartBeat
JobTracker • JobTracker#heartbeat heatbeat TaskTracker • JobTracker#offerService RetireJobs • Job Job • JobTracker#offerService ExpireTrackers • TaskTracker TaskTracker • JobTracker#offerService ExpireLaunchingTasks • 10 ( )heatbeat • RecoveryManager
Hadoopソースコードリーディング MapReduce障害時のフロー
Hadoopソースコードリーディング MapReduce障害時のフロー
Hadoopソースコードリーディング MapReduce障害時のフロー

Recommended

Graduating To Go - A Jumpstart into the Go Programming Language
Graduating To Go - A Jumpstart into the Go Programming LanguageGraduating To Go - A Jumpstart into the Go Programming Language
Graduating To Go - A Jumpstart into the Go Programming Language
Kaylyn Gibilterra
 
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Puppet
 
Airflow Clustering and High Availability
Airflow Clustering and High AvailabilityAirflow Clustering and High Availability
Airflow Clustering and High Availability
Robert Sanders
 
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Jarek Potiuk
 
Getting to Know Airflow
Getting to Know AirflowGetting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
 
Puppet at Spotify (stockholm)
Puppet at Spotify (stockholm)Puppet at Spotify (stockholm)
Puppet at Spotify (stockholm)
Puppet
 
New York GeoConn 2015 Presentation
New York GeoConn 2015 PresentationNew York GeoConn 2015 Presentation
New York GeoConn 2015 Presentation
Matthew Wilson
 
Python Dependency Management - PyconDE 2018
Python Dependency Management - PyconDE 2018Python Dependency Management - PyconDE 2018
Python Dependency Management - PyconDE 2018
Patrick Muehlbauer
 

Recommended

Graduating To Go - A Jumpstart into the Go Programming Language
Graduating To Go - A Jumpstart into the Go Programming LanguageGraduating To Go - A Jumpstart into the Go Programming Language
Graduating To Go - A Jumpstart into the Go Programming Language
Kaylyn Gibilterra
 
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Puppet
 
Airflow Clustering and High Availability
Airflow Clustering and High AvailabilityAirflow Clustering and High Availability
Airflow Clustering and High Availability
Robert Sanders
 
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Manageable data pipelines with airflow (and kubernetes) november 27, 11 45 ...
Jarek Potiuk
 
Getting to Know Airflow
Getting to Know AirflowGetting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
 
Puppet at Spotify (stockholm)
Puppet at Spotify (stockholm)Puppet at Spotify (stockholm)
Puppet at Spotify (stockholm)
Puppet
 
New York GeoConn 2015 Presentation
New York GeoConn 2015 PresentationNew York GeoConn 2015 Presentation
New York GeoConn 2015 Presentation
Matthew Wilson
 
Python Dependency Management - PyconDE 2018
Python Dependency Management - PyconDE 2018Python Dependency Management - PyconDE 2018
Python Dependency Management - PyconDE 2018
Patrick Muehlbauer
 
How to become a Git power user
How to become a Git power userHow to become a Git power user
How to become a Git power user
Deveo
 
Simple Tips and Tricks with Ansible
Simple Tips and Tricks with AnsibleSimple Tips and Tricks with Ansible
Simple Tips and Tricks with Ansible
Keith Resar
 
Whirr devdown
Whirr devdownWhirr devdown
Whirr devdown
Puppet
 
Ember.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.jsEmber.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.js
gavinjoyce
 
Illustrator_Sample
Illustrator_SampleIllustrator_Sample
Illustrator_Sample
Saeid Saadatmand
 
How Openstack is Built
How Openstack is BuiltHow Openstack is Built
How Openstack is Built
Anton Weiss
 
Git in pills : git stash
Git in pills : git stashGit in pills : git stash
Git in pills : git stash
Federico Panini
 
Introducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using itIntroducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using it
Bruno Faria
 
Q&a
Q&aQ&a
Q&a
Subhabrata Mishra
 
Automating Application over OpenStack using Workflows
Automating Application over OpenStack using WorkflowsAutomating Application over OpenStack using Workflows
Automating Application over OpenStack using Workflows
Yaron Parasol
 
Integration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container serviceIntegration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container service
SaltStack
 
Kubernetes London - Musings on Kubernetes Operators
Kubernetes London - Musings on Kubernetes OperatorsKubernetes London - Musings on Kubernetes Operators
Kubernetes London - Musings on Kubernetes Operators
Stuart Leeks
 
AGES Presentation on Web, Python, Django and GeoServer
AGES Presentation on Web, Python, Django and GeoServerAGES Presentation on Web, Python, Django and GeoServer
AGES Presentation on Web, Python, Django and GeoServer
Ng'eno Victor
 
Git
GitGit
Git
IT Booze
 
From airflow to google cloud composer
From airflow to google cloud composerFrom airflow to google cloud composer
From airflow to google cloud composer
Bruce Kuo
 
Weightlifting at SimplySocial
Weightlifting at SimplySocialWeightlifting at SimplySocial
Weightlifting at SimplySocial
Bogdan Gaza
 
Hadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep InsightHadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep Insight
Hanborq Inc.
 
サンプルから見るMap reduceコード
サンプルから見るMap reduceコードサンプルから見るMap reduceコード
サンプルから見るMap reduceコード
Shinpei Ohtani
 
サンプルから見るMapReduceコード
サンプルから見るMapReduceコードサンプルから見るMapReduceコード
サンプルから見るMapReduceコード
Shinpei Ohtani
 
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.comUsing Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
Damien Krotkine
 
BIG DATA ANALYSIS
BIG DATA ANALYSISBIG DATA ANALYSIS
BIG DATA ANALYSIS
Nitesh Singh
 
Quartz Scheduler
Quartz SchedulerQuartz Scheduler
Quartz Scheduler
Software Infrastructure
 

More Related Content

What's hot

Whirr devdown
Whirr devdownWhirr devdown
Whirr devdown
Puppet
 
Ember.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.jsEmber.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.js
gavinjoyce
 
Automating Application over OpenStack using Workflows
Automating Application over OpenStack using WorkflowsAutomating Application over OpenStack using Workflows
Automating Application over OpenStack using Workflows
Yaron Parasol
 

What's hot (16)

How to become a Git power user
How to become a Git power userHow to become a Git power user
How to become a Git power user
 
Simple Tips and Tricks with Ansible
Simple Tips and Tricks with AnsibleSimple Tips and Tricks with Ansible
Simple Tips and Tricks with Ansible
 
Whirr devdown
Whirr devdownWhirr devdown
Whirr devdown
 
Ember.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.jsEmber.js internals backburner.js and rsvp.js
Ember.js internals backburner.js and rsvp.js
 
Illustrator_Sample
Illustrator_SampleIllustrator_Sample
Illustrator_Sample
 
How Openstack is Built
How Openstack is BuiltHow Openstack is Built
How Openstack is Built
 
Git in pills : git stash
Git in pills : git stashGit in pills : git stash
Git in pills : git stash
 
Introducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using itIntroducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using it
 
Q&a
Q&aQ&a
Q&a
 
Automating Application over OpenStack using Workflows
Automating Application over OpenStack using WorkflowsAutomating Application over OpenStack using Workflows
Automating Application over OpenStack using Workflows
 
Integration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container serviceIntegration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container service
 
Kubernetes London - Musings on Kubernetes Operators
Kubernetes London - Musings on Kubernetes OperatorsKubernetes London - Musings on Kubernetes Operators
Kubernetes London - Musings on Kubernetes Operators
 
AGES Presentation on Web, Python, Django and GeoServer
AGES Presentation on Web, Python, Django and GeoServerAGES Presentation on Web, Python, Django and GeoServer
AGES Presentation on Web, Python, Django and GeoServer
 
Git
GitGit
Git
 
From airflow to google cloud composer
From airflow to google cloud composerFrom airflow to google cloud composer
From airflow to google cloud composer
 
Weightlifting at SimplySocial
Weightlifting at SimplySocialWeightlifting at SimplySocial
Weightlifting at SimplySocial
 

Similar to Hadoopソースコードリーディング MapReduce障害時のフロー

サンプルから見るMap reduceコード
サンプルから見るMap reduceコードサンプルから見るMap reduceコード
サンプルから見るMap reduceコード
Shinpei Ohtani
 
Storm distributed processing
Storm distributed processingStorm distributed processing
Storm distributed processing
ducquoc_vn
 
Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014
Ryu Kobayashi
 
London devops logging
London devops loggingLondon devops logging
London devops logging
Tomas Doran
 

Similar to Hadoopソースコードリーディング MapReduce障害時のフロー (20)

Hadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep InsightHadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep Insight
 
サンプルから見るMap reduceコード
サンプルから見るMap reduceコードサンプルから見るMap reduceコード
サンプルから見るMap reduceコード
 
サンプルから見るMapReduceコード
サンプルから見るMapReduceコードサンプルから見るMapReduceコード
サンプルから見るMapReduceコード
 
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.comUsing Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
 
BIG DATA ANALYSIS
BIG DATA ANALYSISBIG DATA ANALYSIS
BIG DATA ANALYSIS
 
Quartz Scheduler
Quartz SchedulerQuartz Scheduler
Quartz Scheduler
 
Map reduce
Map reduceMap reduce
Map reduce
 
Storm distributed processing
Storm distributed processingStorm distributed processing
Storm distributed processing
 
Airflow at WePay
Airflow at WePayAirflow at WePay
Airflow at WePay
 
Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014Treasure Data on The YARN - Hadoop Conference Japan 2014
Treasure Data on The YARN - Hadoop Conference Japan 2014
 
Is hadoop for you
Is hadoop for youIs hadoop for you
Is hadoop for you
 
Big data week presentation
Big data week presentationBig data week presentation
Big data week presentation
 
Introduction to hadoop
Introduction to hadoopIntroduction to hadoop
Introduction to hadoop
 
London devops logging
London devops loggingLondon devops logging
London devops logging
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
 
Devoxx france 2015 influxdb
Devoxx france 2015 influxdbDevoxx france 2015 influxdb
Devoxx france 2015 influxdb
 
Devoxx france 2015 influx db
Devoxx france 2015 influx dbDevoxx france 2015 influx db
Devoxx france 2015 influx db
 
DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜
 
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
OC Big Data Monthly Meetup #5 - Session 1 - AltiscaleOC Big Data Monthly Meetup #5 - Session 1 - Altiscale
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
 
Inside the IT Territory game server / Mark Lokshin (IT Territory)
Inside the IT Territory game server / Mark Lokshin (IT Territory)Inside the IT Territory game server / Mark Lokshin (IT Territory)
Inside the IT Territory game server / Mark Lokshin (IT Territory)
 

More from Ryu Kobayashi

Hadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallHadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 Fall
Ryu Kobayashi
 
Developers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQLDevelopers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQL
Ryu Kobayashi
 
Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra
Hadoopソースコードリーディング第3回 Hadopo MR + CassandraHadoopソースコードリーディング第3回 Hadopo MR + Cassandra
Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra
Ryu Kobayashi
 

More from Ryu Kobayashi (6)

PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core enginePLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
PLAZMA TD Tech Talk 2018 at Shibuya: Hive2 as a new td hadoop core engine
 
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 WinterHuahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
Huahin Framework for Hadoop, Hadoop Conference Japan 2013 Winter
 
Hadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallHadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 Fall
 
Developers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQLDevelopers summit cassandraで見るNoSQL
Developers summit cassandraで見るNoSQL
 
Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra
Hadoopソースコードリーディング第3回 Hadopo MR + CassandraHadoopソースコードリーディング第3回 Hadopo MR + Cassandra
Hadoopソースコードリーディング第3回 Hadopo MR + Cassandra
 
AWSを使ったトラッキングログ収集
AWSを使ったトラッキングログ収集AWSを使ったトラッキングログ収集
AWSを使ったトラッキングログ収集
 

Recently uploaded

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Delhi Call girls
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Igalia
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Malak Abu Hammad
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Enterprise Knowledge
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Enterprise Knowledge
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Delhi Call girls
 

Recently uploaded (20)

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 

Hadoopソースコードリーディング MapReduce障害時のフロー

  • 1. 2 Hadoop MapReduce
  • 2. • • Twitter @ryu_kobayashi • ...
  • 3.
  • 4. • MapReuceTaskTracker JobTracker
  • 5. • MapReuceTaskTracker JobTracker • Hadoop P176-178
  • 6. • MapReuceTaskTracker JobTracker • Hadoop P176-178 •
  • 7.
  • 8. •
  • 9. • • Ubuntu Debian
  • 10. • • Ubuntu Debian • Ubuntu IPv6 OS -Djava.net.preferIPv4Stack=true
  • 11. • • Ubuntu Debian • Ubuntu IPv6 OS -Djava.net.preferIPv4Stack=true • OS •
  • 13. • JobClient#runJob • JobClint#monitorAndPrintJob mapred-site.xml jobclient.output.filter FAILDE monitorAndPrintJob Job
  • 14. TaskTracker • TaskTracker#run • TaskTracker#offerService • TaskTracker#markUnresponsiveTasks Task • JobTracker#statusUpdate • TaskInProgress#reportProgress • JobTracker TaskTracker#offerService HeartBeat
  • 15. JobTracker • JobTracker#heartbeat heatbeat TaskTracker • JobTracker#offerService RetireJobs • Job Job • JobTracker#offerService ExpireTrackers • TaskTracker TaskTracker • JobTracker#offerService ExpireLaunchingTasks • 10 ( )heatbeat • RecoveryManager

Editor's Notes