Building Data Products using Hadoop at Linkedin - Mitul Tiwari
•Download as PPT, PDF•
13 likes•2,011 views
Hadoop and other big data tools such as Voldemort, Azkaban, and Kafka, drive many data driven products at LinkedIn such as “People You MayKnow” and various recommendation products such as “Jobs You May Be Interested In”. Each of these products can be viewed as a large scale social recommendation problems, which analyzes billions of possible options, and suggest appropriate recommendation. Since these products analyzes billions of edges and terabytes of data daily, it can be built only using a large scale distributed compute infrastructure. Kafka publish-subscribe messaging system is used to get the data in Hadoop file system. Hadoop MapReduce is used as the basic building block to analyze billions of potential options, and predict recommendation. Over a hundred MapReduce tasks are combined together in a work-flow uising Azkaban, a Hadoop work-flow management tool. The output of Hadoop jobs is finally stored in Voldemort key-value store to serve the data at run-time for efficiency. During this talk audience will get a basic understanding of link prediction problem behind “ People You May Know” feature, which is a large scale social recommendation problem. Overview of the solution of this problem using Hadoop MapReduce, Azkaban workflow management tool, and Voldemort key-value store will be presented. I will also describe how to efficiently compute the number of common connections (triangle closing) using Hadoop Mapreduce, which is one of the many signals in link prediction. Overall, people interested in building interesting applications using Hadoop MapReduce will hugely benefit from this talk.
![Building Data Products using Hadoop at Linkedin ,[object Object],[object Object],[object Object]](https://image.slidesharecdn.com/bigdatalinkedin-110813082617-phpapp01/85/Building-Data-Products-using-Hadoop-at-Linkedin-Mitul-Tiwari-1-320.jpg)
Recommended
Recommended
More Related Content
More from BigDataCloud
More from BigDataCloud (20)
Recently uploaded
Recently uploaded (20)
Building Data Products using Hadoop at Linkedin - Mitul Tiwari
- 2. Who am I?
- 4. People You May Know
- 6. Viewers of this profile also ...
- 7. Skills
- 8. InMaps
- 18. People You May Know Alice Bob Carol How do people know each other?
- 19. People You May Know Alice Bob Carol How do people know each other?
- 20. People You May Know Alice Bob Carol Triangle closing How do people know each other?
- 21. People You May Know Alice Bob Carol Triangle closing Prob(Bob knows Carol) ~ the # of common connections How do people know each other?
- 22. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 24. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 25. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 26. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 27. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 28. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
- 34. Our Workflow triangle-closing top-n
- 35. Our Workflow triangle-closing top-n push-to-prod
- 37. Our Workflow triangle-closing top-n push-to-prod
- 38. Our Workflow triangle-closing top-n push-to-prod remove connections
- 39. Our Workflow triangle-closing top-n push-to-prod remove connections push-to-qa
- 40. PYMK Workflow
- 44. Azkaban Workflow
- 45. Azkaban Workflow
- 46. Azkaban Workflow
- 47. Our Workflow triangle-closing top-n push-to-prod remove connections
- 48. Our Workflow triangle-closing top-n push-to-prod remove connections
- 52. Data Cycle
- 54. Our Workflow triangle-closing top-n push-to-prod remove connections
- 61. Questions?
Editor's Notes
- Hi, I am Mitul Tiwari. Today I am going to talk about building data driven products using Hadoop at LinkedIn.
- I am part of Search, Network, Analytics team at LinkedIn, and I work on data driven products such as People You May Know.
- let me illustrate through a few examples of data products at LinkedIn
- LinkedIn is the second largest social network for professionals with more than 100 million members. PYMK is a large scale recommendation system that helps you connect with others. Basically, PYMK is a link prediction problem, where we analyze billions of edges to recommend possible connections to you. A big big-data problem!
- Another example of a data product at LinkeIn is “Profile Stats” or “Who Viewed My Profile”. Profile Stats provides analytics about your profile on LinkedIn. It provides stats about who viewed your profile, what are the top search queries leading to your profile, the number of profile views per day/week, location of the visitor, etc. We have billions of pageviews per month, Profile Stats is another big data problem.
- Another example of Data Product at LinkedIn is “Viewers of this profile also viewed these profiles”. A collaborative filtering way of suggesting profile.
- topic pages for skills
- Visualize your connections. Cluster your connections based on their connection density among them.
- The key ideas behind these data products are Recommendations, Analytics, Insight, and Visualization.
- Some challenges behind building data driven products at linkedin. A naive implementation of PYMK may result in generating 120mX120m pairs, which is 14400 trillion pairs. So you have to be smart about it. So which data product would you like me to build during this talk?
- Here is a pig script to do triangle closing, that is, find the number of common connections between any pair of members.
- So how many of you are familiar with Pig? Let me refresh some Pig constructs for those who are not very familiar with Pig.
- First you load connections data that is in bidirectional pairs format. Representing each direction of an edge by a pair of member ids.
- Then we group connections pairs by source_ids to aggregate all connections for each member.
- From aggregated connections we generate pairs of members (id1, id2) which are friend-of-friend through a source_id
- Now we group by (id1, id2) to aggregate all common connections, and count to find the number of common connections.
- Finally, we store common connections data in HDFS.
- Let me illustrate the triangle closing through our running example. First, we load each direction edge represented by a pair.
- Then we group connections pairs by source_ids to aggregate all connections for each member.
- From aggregated connections we generate pairs of members (id1, id2) which are friend-of-friend through a source_id
- Finally we group by (id1, id2) to aggregate all common connections, and count to find the number of common connections.
- After we are done with triangle closing we can list each members’ friends-of-friends ordered by the number of common connections. .
- Since there might be too many people who are your friends-of-friends, you might want to select top n from that list. For example, there are more than hundred-fifty thousands people who are my friends-of-friends
- Next you need to push this data out in production. So that’s a simple workflow.
- I just described how you can build People You May Know by doing triangle-closing and finding out friends-of-friends, and the number of common connections between them. As you just saw there might be multiple jobs dependent on each other that you have to run in that order. So how do we manage our workflow?
- We need to ensure that we are showing good quality data to our members. First, we verify that data transfer between HDFS and production system is done properly. Second, we push data to a QA store with a viewer to check any blatant mistakes. Third, we have can explain any PYMK recommendation, how and why that recommendation is appearing. Fourth, we have ways to rollback in case something goes wrong. And finally, we have unit tests in place to check things are processes as we desire
- First, we can improve performance by 50% by utilizing symmetry in our triangle-closing. If Bob is a friend-of-friend of Carol then Carol is a friend-of-friend of Bob. Second, there are supernodes in our social graph. For examples, Barack Obama has more than 10000 connections on LinkedIn. If we generate f-o-f pairs from his connections involving Barack Obama, there will be more 100 million pairs of f-o-f. Third, we can sample certain number of connections to decrease the the number of pairs from f-of-f, and randomize so that we generate different pairs every day.
- Oscon Data award for open source contributions.