Honu - A Large Scale Streaming Data Collection and Processing Pipeline__HadoopSummit2010
•
26 likes•3,368 views
Hadoop Summit 2010 - Developers Track Honu - A Large Scale Streaming Data Collection and Processing Pipeline Jerome Boulon, Netflix
![Honu: A large scale data collection and processing pipeline ,[object Object],Netflix](https://image.slidesharecdn.com/7honuhadoopsummit2010-100630134425-phpapp02/85/Honu-A-Large-Scale-Streaming-Data-Collection-and-Processing-Pipeline__HadoopSummit2010-1-320.jpg)
![[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Session Agenda](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Honu - A Large Scale Streaming Data Collection and Processing Pipeline__HadoopSummit2010
Similar to Honu - A Large Scale Streaming Data Collection and Processing Pipeline__HadoopSummit2010 (20)
More from Yahoo Developer Network
More from Yahoo Developer Network (20)
Recently uploaded
Recently uploaded (20)
Honu - A Large Scale Streaming Data Collection and Processing Pipeline__HadoopSummit2010
- 5. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 6. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 7. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 8. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 9. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 10. Current Netflix deployment EC2/EMR Applications S3 Hive MetaStore Hive & Hadoop EMR clusters Hive & Hadoop EMR (for ad-hoc query) Honu Collectors AMAZON EC2
- 14. Structured Log API – Using Annotation @ Resource (table=”MyTable") public class MyClass implements Annotatable { @ Column ("movieId") public String getMovieId() { […] } @Column("clickIndex") public int getClickIndex() { […] } @Column("requestInfo") public Map getRequestInfo() { […] } } DB=MyTable Movied=XXXX clieckIndex=3 requestInfo.returnCode=200 requestInfo.duration_ms=300 requestInfo.yyy=zzz log.info(myAnnotatableObj);
- 15. Structured Log API - Key/Value API KeyValueSerialization kv ; kv = new KeyValueSerialization(); […] kv.startMessage("MyTable"); kv.addKeyValue("movieid", ”XXX"); kv.addKeyValue("clickIndex", 3); kv.addKeyValue(”requestInfo", requestInfoMap); DB=MyTable Movied=XXXX clickIndex=3 requestInfo.returnCode=200 requestInfo.duration_ms=300 requestInfo.yyy=zzz log.info(kv.generateMessage());
- 17. Data Processing pipeline Application Collector Hive Data collection pipeline Data processing pipeline M/R
Editor's Notes
- This is the Title slide. Please use the name of the presentation that was used in the abstract submission.
- This is the agenda slide. There is only one of these in the deck.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is the final slide; generally for questions at the end of the talk. Please post your contact information here.