The “People You May Know” (PYMK) recommendation service helps LinkedIn’s members identify other members that they might want to connect to and is the major driver for growing LinkedIn's social network. The principal challenge in developing a service like PYMK is dealing with the sheer scale of computation needed to make precise recommendations with a high recall. PYMK service at LinkedIn has been operational for over a decade, during which it has evolved from an Oracle-backed system that took weeks to compute recommendations to a Hadoop backed system that took a few days to compute recommendations to its most modern embodiment where it can compute recommendations in near real time.
This talk will present the evolution of PYMK to its current architecture. We will focus on various systems we built along the way, with an emphasis on systems we built for our most recent architecture, namely Gaia, our real-time graph computing capability, and Venice our online feature store with scoring capability, and how we integrate these individual systems to generate recommendations in a timely and agile manner, while still being cost-efficient. We will briefly talk about the lessons learned about scalability limits of our past and current design choices and how we plan to tackle the scalability challenges for the next phase of growth.
https://qcon.ai/qconai2019/presentation/people-you-may-know-fast-recommendations-over-massive-data
[QCon.ai 2019] People You May Know: Fast Recommendations Over Massive Data
1. People You May Know
Fast Recommendations Over Massive Data
Jeff Weiner
Chief Executive Officer
Sumit Rangwala
Artificial Intelligence
Felix GV
Data Infrastructure
3. My Professional Network
Professional network in real world Professional network on LinkedIn
Sumit Felix
Peter
Amol
Gaojie
Sumit Felix
Peter
Amol
Gaojie
4. My Professional Network
Professional network in real world Professional network on LinkedIn
Sumit Felix
Peter
Amol
Gaojie
Sumit Felix
Peter
Amol
Gaojie
Predicting
real world
connections
5. Helps grow member’s professional network
Recommends people that one might know
People You May Know
Enables many other LinkedIn services
6. Talk Outline
People You May Know
PYMK: Generating Recommendations
PYMK Architecture Evolution
PYMK Rebirth
Insights and Road Ahead
8. PYMK: Prediction Strategy
Data Mining
• LinkedIn’s Economic Graph
• Member’s activities and profile
LinkedIn Economic Graph
Sumit Felix
Peter
Amol
Gaojie
9. PYMK: Prediction Strategy
Data Mining
• LinkedIn’s Economic Graph
• Member’s activities and profile
LinkedIn Economic Graph
Felix
Peter
Amol
Gaojie
Microsoft
USC
Sumit
11. PYMK: Candidate Generation
Using commonalities in
economic graph
• Friends of my friends
(triangle closing)
LinkedIn Economic Graph
Amol
Peter Gaojie
Sumit Felix
12. PYMK: Candidate Generation
Using commonalities in
economic graph
• Friends of my friends
(triangle closing)
• Coworkers
• Personalized Page Rank
LinkedIn Economic Graph
Amol
Peter Gaojie
Felix
Microsoft
Sumit
13. PYMK: Feature Generation
Using economic graph
characteristics
• Number of common friends
Using member
activities/profile
• Common work location
LinkedIn Economic Graph
Amol
Peter Gaojie
Felix
Microsoft
Sumit
19. PYMK: The Beginning
Problem Space
• 10s of millions of members
Architecture
• Pre-compute using SQL
Shortcomings
• Staleness of 6 weeks to 6 months
• Extraneous computation
Oracle
20. PYMK: The Beginning
Problem Space
• 10s of millions of members
Architecture
• Pre-compute using SQL
Shortcomings
• Staleness of 6 weeks to 6 months
• Extraneous computation
Oracle PYMK
Service
Online service request
21. PYMK: Keeping up with Growth
Problem space
• Low 100s of millions of members
Architecture
• Pre-compute using Hadoop MR
• Push to a key-value store
Shortcomings
• Staleness of 2-3 days
• Extraneous computation
Voldemort
PYMK
Service
22. PYMK: Pushing the Technology Limits
Problem Space
• Mid 100s of millions of members
Architecture
• Pre-compute using Spark1
• Push to a key-value store
Shortcomings
• Staleness of 1-2 days
• Excessive computation cost
Venice
[1] Managing Exploding Big Data
PYMK
Service
23. PYMK: Exploring Data Freshness
Problem Space
• Use up to date member data
Architecture
• Hybrid offline-online approach
Shortcomings
• Split-brain design
• Didn’t scale
Venice
Realtime signals
PYMK
Service
33. Gaia: Overview
Gaia
Any kind of graph
Updates to graph
Graph algorithm code
A snapshot
on HDFS
Via Kafka, etc.
Using
compute
framework
e.g., triangle closing,
random graph walks
52. At a glance
Server-side Computation
• Simple vector operations
• Smaller response size
• Big input (vector)
• Small output (scalar)
• Declarative API
• No arbitrary code
53. More tuning
Fast Avro
• Online feature retrieval
• 60 to 40 ms (p99)
• Embeddings w/ computation
• 60 to 35 ms (p99)
• Now open-source!
• github.com/linkedin/avro-util
55. PYMK: Recommendation System
Candidate
Generation
Sumit might know Amol’s friend, Felix
Sumit and Felix have one common friend
Sumit and Felix both work in Bay Area
PYMK Service
Feature
Generation
Scoring Sumit and Felix likely know each other
Venice
Gaia
56. PYMK: Today
Venice
PYMK
Service
Gaia
1. Ingest in
Gaia & Venice
2. Candidate gen
& graph features
from Gaia
4. Final scoring
by PYMK Service
3. Member features
& partial scoring
from Venice
Staleness
• Seconds to minutes
57. Key Learnings
• Pre-computation is viable for many products
• Scaling RT computation requires moving compute close to data
• Infra aware Machine Learning
59. • Further scale Gaia & Venice
• More candidates
• More features
• Larger features
• More complex computations
ML-Aware Infra
60. • Continue democratizing access
• Easier onboarding to Venice & Gaia
• Multi-tenancy for Venice Compute
• Integration with other frameworksProductive ML