Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Managing an Experimentation Platform by LinkedIn Product Leader

Main Takeaways:

-Establishing a culture of experimentation at scale
-Developing the product vision and strategy
-Backlog prioritization based on Impact Score formula

  • Be the first to comment

  • Be the first to like this

Managing an Experimentation Platform by LinkedIn Product Leader

  1. 1. Managing an Experimentation Platform by LinkedIn Product Leader
  2. 2. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™
  3. 3. Corporate Training Level up your team’s Product Management skills
  4. 4. Free Product Management Resources BOOKS EVENTS JOB PORTAL COMMUNITIES COURSES
  5. 5. Makram Mansour, PhD [Product Leader @ LinkedIn] Managing an Experimentation Platform
  6. 6. Makram Mansour Product Manager, LinkedIn 1 LinkedIn Profile 2 Education MS & PhD in EE from UIUC Stanford LEAD from Stanford GSB 3 Product @ LinkedIn LinkedIn Experimentation & Data Insights 4 Product @ Texas Instruments Online Design Tools 5 Engineer @ Intel Intel Server Chipsets 6 Values Out-of-the-box thinker who is not afraid of taking risks, gets more committed when people tell me “it cannot be done”, and strongly believe in the saying: “where there’s a will, there’s a way”
  7. 7. 1 2 3 4 Agenda Experimentation @ LinkedIn Backlog Prioritization Workflow Optimization Final note on Vision & Strategy
  8. 8. Experimentation @ LinkedIn
  9. 9. Create economic opportunity for every member of the global workforce.
  10. 10. Connect the world’s professionals to make them more productive and successful.
  11. 11. The Economic Graph 740M 57M 14M 38K 120K 280B Members Companies Jobs Skills Schools Knowledge
  12. 12. LinkedIn’s data in motion 100B graph edges, 2B nodes 650K graph queries/second 1,500PB total data storage 11T messages/day on Kafka 25B model parameters in AI models 208M contributions/day 30B Feed Updates viewed/month 750PB average daily storage on Hadoop 12B page views/day 180M messages sent/day 96M profile actions/day
  13. 13. Complex growth engine full of “network” effects One thing we learnt over the years is that even small, localized changes can have massive impact! Maintaining and accelerating growth requires a strong discipline around experimentation and data.
  14. 14. Cost of not doing A/B testing can be high! Profile top ads 5-pixel height decrease CTR drop on 11/11/2013
  15. 15. We test everything Frontend, ranking algorithms, and backend infra Experimentation activity • 100/day new tests • 400/day tests ramps • 200/week AI/ML models Experimentation adoption • 5000+ experiment owners • 2000+ WAU • 41k tests running simultaneously Infrastructure activity • 2M QPS • 35T/day evaluations
  16. 16. 20k Metrics 8k A/B testable Metrics Single source-of-truth Self-served, custom metrics 90k QPS 2.5PB/day 20T/day records processed on Hadoop Tracking Events Kafka Offline Infrastructure Unified Metrics Platform Visualization Experimentation Anomaly Detection & Root Cause Analysis Massive experimentation and metrics data
  17. 17. 1. Targeting Helps run experiments on different audience attributes: - Location - Job title - Company - Industry - Education - Language - Device - Connections - etc. 2. Ramping Helps you easily and safely ramp/de-ramp a feature over time. Provides standard framework to assist users to ramp with the right balance of Speed, Quality, and Risk 3. EXperimentation Advanced experimentation infrastructure built for large-scale use. Some of the features: - Multivariate testing - Advanced randomization - Metrics Reporting & Alerting - Insights for ease of interpreting results - Variance Reduction Methods - Most Impactful Experiments TREX is LinkedIn’s Experimentation Platform It’s a unified platform for Targeting, Ramping, and EXperimentation
  18. 18. Backlog Prioritization
  19. 19. 1 2 3 4 Prioritization Framework Quantifies business value Allow “apples-to-apples” stack-rank and prioritization Makes decision-making data-driven and transparent Sets clear guidelines on what data is needed for every ask
  20. 20. Prioritization Framework Four Pillars Value Measured by impact to Site Up, Productivity, Revenue, Key Metrics, or User Satisfaction Leverage Measured by # of users impacted in 12 months or less Urgency Measured by urgency: blocked, impaired but have short-term workaround, or inconvenient Cost Measured in engineer-quarters (engineers needed * no. of quarters to complete)
  21. 21. Computing Value (V) Rubric Select up to two categories and multiply their corresponding numbers E.g., V1 = G = 4, V2 = P = 2 ⇒ V = 4 * 2 = 8 GCNs (G) Productivity (P) Revenue (R) Metrics (M) User Satisfaction (S) GCN Savings: 1. 1 minor GCN/qrtr 2. 2-4 minor GCNs 3. 1-2 medium GCNs 4. 1+ major GCNs Savings Criteria: 1. < 1hr /usr/mnth 2. 2 - 5 hrs saved 3. 6 - 10 hrs saved 4. > 10 hrs saved Revenue Lift: 1. < $1M annually 2. $1M - $10M 3. $10M - $30M 4. > $30M Key Metrics Lift: 1. < 0.5% lift 2. 0.5% - 1.0% 3. 1.0% - 2.0% 4. > 2.0% NSAT Score Lift: 1. < +0.2 lift 2. +0.3 - +0.5 3. +0.6 - +0.9 4. > +1.0
  22. 22. Computing other Rubrics T-REX Users Impacted 1. < 100 users 2. 100 - 499 3. 500 - 1999 4. > 2000 Leverage (L) User’s work impact Costing Formula Urgency (U) Cost (C) 1. Inconvenient 2. Impaired; workaround in place 3. Blocked 1. # of Engineers 2. # of Quarters 3. C = E * Q
  23. 23. Prioritization - Final Scoring Guidelines Final Ranking Criteria 03 ● Rank by largest Impact Score ● Override for large ROIs (Quick Wins) Calculate ROI 02 ROI = Impact Score / Cost Calculate Impact Score (IS) 01 Impact Score = Leverage * Value * Urgency
  24. 24. Ideal Mix 40% 30% 20% 10% Big Bets Home Run Quick Wins Fall Back Level of Effort Ranking
  25. 25. Workflow Optimization
  26. 26. Optimize your Workflow Prioritized 50% Power Design Sign-off Plan Sign- off Feature Request Feature Release Code Ready Ideate Flow diagram, Workflows, Design, Use cases Ramp Verify solution, fix unforeseen issues, capture metrics, Derive success stories Announce Announce highlighting features and impact. Include user testimonials, training, and documentation Learn Problem statements, User stories, Impact assessment, Requirements gathering, Prioritization, PRD Design RFC, Hi-Fi Design, Acceptance test cases, WBS, Timeline Build Modular code, Bug bashes, Unit & Integration testing, Previews Design Thinking Approach with Experimentation in the Core of your Purpose Gain insights in what our customers really want and why. Build clarity in terms of Before / After state. Gradual release ramp to efficiently address unforeseen issues. Measure success metrics and announce with impact!
  27. 27. Sample PRD Areas of focus: Problem Statement Core Use Cases Prioritization Details Objectives and Key Results Before and After Milestones Sign-offs
  28. 28. Note on Vision and Strategy
  29. 29. Be clear on your Vision to Values Vision • The Dream. The Future. What you inspire your product will impact and become. Mission • Briefly describes the goals and purpose of your product. Why the product exists? Target Audience • List down your primary user personas • Prioritize them. Also be clear on non-users. • Group them in Producers / Consumers if you have a platform product Strategy • Series of strategic objectives and roadmap milestones Priorities • Stack ranked list of critical initiatives. “If we can only do one thing this quarter, what would it be?” Objectives (Key Metrics) • True North Metrics • Sign-Post Metrics • Guardrail Metrics Values • Your guiding principles in making day-to-day decisions
  30. 30. Thank you
  31. 31. Part-time Product Management Training Courses and Corporate Training