Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lean Analytics for Startups and Enterprises


Published on

Latest Lean Analytics workshop from the Lean Startup Week in San Francisco. Focusing on what metrics matter to both startups and big corporations. Incorporates elements of corporate innovation into the Lean Analytics framework to help bigger companies think through the data that really matters.

Published in: Business

Lean Analytics for Startups and Enterprises

  1. 1. Using Lean Analytics for Startups and Enterprises Ben Yoskovitz | @byosko
  2. 2. Introduction @byosko I am a product guy entrepreneur author angel investor
  3. 3. Find me online Blog: Slideshare: Book: Email: @byosko
  4. 4. CORPORATE PARTNERS VENTURE-BACKABLE FOUNDERS PRE-SEED FUNDING BETTER STARTUPS + + = Highline BETA is a startup co-creation company that launches new ventures with leading corporations. @byosko
  5. 5. Metrics: The Fundamentals
  6. 6. Metrics: The fundamentals ● How data fits in ● What makes a good metric ● Types of metrics ● Analytical superpowers @byosko
  7. 7. How to get things built properly (in theory)
  8. 8. Everyone has great ideas, right? People love this part (but that’s not always a good thing!) This is where things start to fall apart. No data, no learning. Build Measure Learn seems so easy!
  9. 9. INTELLECTUALLY HONESTY Follow the Lean model and it becomes increasingly hard to lie, especially to yourself.
  10. 10. FOCUS Don’t chase shiny objects. You might succeed without focus, but it’ll be by accident.
  11. 11. BETTER DECISION MAKING Everyone has data. The key is figuring out what pieces will improve your learning and decision making.
  12. 12. USE YOUR GUT PROPERLY Instincts are experiments. Data is proof.
  13. 13. So what makes a good metric?
  14. 14. Question: What are the metrics you’re tracking? ● Take 2 minutes to write down the key metrics you’re tracking (or your business is tracking) right now. ● These could be at a business level or project level. ● At the end of this section we can re-evaluate if the metrics you’re tracking are still the right ones. @byosko
  15. 15. WHAT IS ANALYTICS? Analytics is the measurement of movement towards business goals.
  16. 16. A good metric is: Understandable If you’re busy explaining the data, you won’t be busy acting on it. Comparative Active Users vs. Active Users/ month Ratio / Rate % Monthly Active Users Behavior Changing You’ll know how you’ll change your business based on what the metric tells you. @byosko
  17. 17. If a metric won’t change how you behave, it’s a bad metric. THE GOLDEN RULE OF METRICS
  18. 18. Acquisition1-15% Low cost of acquisition, high checkout Customers that buy >1x in 90d Then you are in this mode Your customers will buy from you You are just like Focus on 15-30% >30% Hybrid Loyalty Once 2-2.5 >2.5 per year per year 70% 20% 10% of retailers of retailers of retailers Increasing return rate, market share Loyalty, selection, inventory size (Thanks to Kevin Hillstrom for this.) Metrics help you know yourself:
  19. 19. Types of Metrics
  20. 20. Vanity vs. Actionable metrics Vanity Actionable Makes you feel good but doesn’t change how you’ll act. Helps you pick a direction and change your behavior. “Up and to the right.” These are good. @byosko
  21. 21. Beware of vanity metrics: Users Follows / friends / likes Logins This tells you nothing about what they did, why they stuck around, or why they left. Count actions instead. Count how many followers will do your bidding. What are they actually doing when they login? Logins don’t tell you about actions and value. Downloads Sure, people need to download your app in order to use it, but so what? @byosko
  22. 22. The best (worst!) vanity metric of all time… # of Features @byosko
  23. 23. Qualitative vs. Quantitative metrics Qualitative Quantitative Unstructured, anecdotal, revealing, hard to aggregate. Numbers and stats; hard facts, but less insights. Warm and fuzzy. Cold and hard. @byosko
  24. 24. Discover qualitatively. Prove quantitatively.
  25. 25. Do Airbnb hosts get more business if their property is professionally photographed?
  26. 26. Gut instinct (hypothesis) Professional photography helps Airbnb’s business Concierge MVP Sent 20 photographers out into the field Measure the results Compared photographed listings to control group Make a decision Launched photography as a new feature to all hosts CASE STUDY Do professional photos make a difference?
  27. 27. Exploratory vs. Reporting metrics Exploratory Reporting Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages. Predictable. Keeps you abreast of normal, day-to- day operations. Can be managed by exception. Cool. Necessary. @byosko
  28. 28. ! Started as Circle of Friends ! Leveraged Facebook early ! Grew to 10M users fast ENGAGEMENT SUCKED! CASE STUDY Finding insights in the data
  29. 29. ENGAGEMENT SOLVED. CASE STUDY Moms are crazy! (in a good way) ! Messages to one another were ~50% longer ! 115% more likely to attach a picture to a post ! 110% more likely to engage in a threaded conversation ! Invited friends were 50% more likely to become engaged users ! 60% more likely to accept invitations to the app
  30. 30. Lagging vs. Leading metrics Lagging Leading Historical metric that shows you how you’re doing: reports the news. Number today that shows a metric tomorrow: makes the news. Start here. Try and get here. @byosko
  31. 31. Examples of leading metrics A Facebook user reaching 7 friends within 10 days of signing up. (Chamath Palihapitiya) A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi Wang) A Twitter user who follows a certain number of people, and a certain percentage of those people follow the user back. (Josh Elman) A LinkedIn user getting to X connections in Y days. (Elliot Schmukler) @byosko
  32. 32. 1. People who install the Chrome extension 2. People who connect more than 1 social account 3. People who share 15 pieces of content in 7 days CASE STUDY Buffer discovered 3 leading metrics
  33. 33. Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Correlation vs. causation
  34. 34. Correlated vs. Causal metrics Correlated Causal Two variables that are related (but may be dependent on something else.) An independent variable that directly impacts a dependent one. Ice cream and drowning. Summertime and drowning / Summertime and eating ice cream @byosko
  35. 35. A leading, causal metric is a superpower.
  36. 36. Causality is a superpower because it lets you change the future. Correlation lets you predict the future Causality lets you change the future “I will have 420 engaged users and 75 paying customers next month.” “If I can make more first time visitors stay for 17 minutes I will increase sales in 90 days.” Pick a metric to change Find correlation Test for causality Optimize the causal factor @byosko
  37. 37. Cohort analysis @byosko
  38. 38. Ricky (product manager) has some ideas for improving the “Proposal Send Screen” (based on qualitative feedback & his gut), but before prioritizing this work, he digs into the data. Putting basic data to use
  39. 39. 50% of people send proposals through Proposify (50% don’t) (quantitative) — Is this good or bad? Putting basic data to use
  40. 40. Ricky isn’t sure. So he’s going to need to look at additional data (exploratory): • Churn • Proposal won rate • Any correlations here? Putting basic data to use
  41. 41. @byosko • Also needs to do more direct customer development to learn more (qualitative) • All of this work might lead to additional, meaningful product dev (actionable) Putting basic data to use
  42. 42. Look back at the metrics you’re tracking ● Remember the metrics you wrote down earlier? How do they stack up now? Are they good metrics? ● What might you change about the metrics you’re tracking as a business and/or on a project/feature level? @byosko
  43. 43. Quick summary on the basics of analytics ● Analytics is about measuring movement towards business goals ● Analytics is about simplifying not complicating ● Analytics is about helping you focus on what really matters ● Remember the Golden Rule: A good metric has to change your behaviour @byosko
  44. 44. Measuring Success: An introduction to Lean Analytics
  45. 45. Lean Analytics Framework ● The five stages of business & product development ● Mapping business models ● The One Metric That Matters (KPIs) ● The Lean Analytics Cycle @byosko
  46. 46. Two keys: the Business you’re in & the Stage you’re at What business are you in? What stage are you at? ! E-Commerce ! SaaS ! Free Mobile App ! 2-Sided Marketplace ! Media ! User-Generated Content ! Empathy ! Stickiness ! Virality ! Revenue ! Scale @byosko
  47. 47. Big companies need one more thing. An understanding of what type of innovation they’re doing.
  48. 48. Core Adjacent Transformative Do the same thing better. Nearby product, market, or method. Start something entirely new. Regional
 optimizations. Innovation, go-to-market strategies. Reinvent the business model. • Get there faster • Smaller batches • Solution, then testing • Increased accountability • Customer development • Test similar cases • Parallel deployment • Analytics & cycle time • Fail fast • Skunkworks/R&D • Focus on the search • Ignore the current model & margins Many models for enterprise innovation
  49. 49. Know the problem (customers tell you it) Know the solution (customers/regulations/ norms dictate it.) Know the problem (market analysis) Don’t know the solution (non-obvious innovation confers competitive advantage.) Don’t know the problem (just an emerging need/change) Don’t know the solution. Waterfall:
 Execution matters Agile/scrum:
 Iteration matters Lean Startup: Discovery matters Another way to look at it Core Adjacent Transformative
  50. 50. Current
 state Business optimization Product,
 method innovation
 Business model innovation
 You can convince executives of this because some of it is familiar. This terrifies them because it eats the current business. A three-maxima model for enterprise innovation
  51. 51. Improvement Adjacency Remodelling Do the same,
 only better. Explore what’s
 nearby quickly Try out new
 business models Lean approaches apply, but the metrics vary widely. Sustain / core Innovate / adjacent Disrupt / transformative
  52. 52. Sustaining Adjacent Disruptive Next year’s car Electric car,
 same dealer On-demand, app-based
 car service
  53. 53. So the metrics that matter to a big company are dependent on the type of innovation being done.
  54. 54. Stages of business & product development
  55. 55. Eric’s three engines of growth Stickiness Virality Price Approach Math that matters Keep people coming back. Get customers faster than you lose them. Make people invite friends. How many they tell, how fast they tell them. Spend money to get customers. Customers are worth more than they cost. @byosko
  56. 56. Dave McClure’s Pirate Metrics
  57. 57. Dave McClure’s Pirate Metrics Acquisition Activation Retention Referral Revenue How do your users become aware of you? Do drive-by visitors subscribe, use, etc.? Does a one time user become engaged? Do users promote your product? Do you make money from user activity?
  58. 58. The Lean Analytics Stages Empathy You’ve found a real, poorly-met need that a reachable market faces. You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for. Your users and features fuel growth organically and artificially. You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem. STAGE GATE Stickiness Virality Revenue Scale
  59. 59. The Lean Analytics Stages Empathy You’ve found a real, poorly-met need that a reachable market faces. You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for. Your users and features fuel growth organically and artificially. You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem. STAGE GATE Stickiness Virality Revenue Scale Most products (and startups) fail at this point.
  60. 60. CASE STUDY ! Stage: Empathy/Stickiness ! Model: E-Commerce ! Originally tied to Instagram with an “Insta-Order” feature Jumping the gun on product development
  61. 61. Optimize for 1st time purchases or repeat orders? WITH INSTA-ORDER Click checkout Confirmation page Confirm order Success page Sign in to PayPal Back to PayPal Authorized pre-approved payments WITHOUT INSTA-ORDER Click checkout Sign in to PayPal Confirmation page Confirm order Success page ● 2x transactions ● Lower bounce rate ● Sign-in goals increased
  63. 63. Mapping business models
  64. 64. Does recurring revenue work for everyone? CASE STUDY @byosko
  65. 65. The leader in predictive analytics for people. Clearfit helps thousands of companies build better teams. As featured in: CASE STUDY 10x revenue increase off of 3x in sales volume “People don’t do subscriptions for haircuts, hamburgers or hiring. You have to understand your customer, who they are, how and why they buy, and how they value your product or service.” - Ben Baldwin
  66. 66. The goal is to understand the customer’s lifecycle / journey through every touchpoint with your product.
  67. 67. Paid Direct WOM Search Inherent virality Customer Acquisition Cost VISITOR User FORMER USERS Engaged user Reactivate Trial over Invite others Paying customer Disengaged Account cancelled Freemium / trial offer Enrollment Disengaged user Cancel Cancel Reactivate FORMER CUSTOMERS Billing info exp. Resolution Dissatisfied Capacity Limit Upselling Signup conversion rate Free user disengagement Freemium churn Reactivation rate User lifetime value Customer lifetime value Trial abandonment rate DAU/WAU/MAU Paid conversion Viral coefficient Viral rate Paid churn rate Support data Tiering Upselling rate SaaS Customer Lifecycle
  68. 68. Returning Paid Direct Search Viral Customer Acquisition Cost VISITOR E-Commerce Customer Lifecycle Navigation Search Reco Engine 1-time buyer Cart Additions Conversion Logistics, delays Delivery Enrollment Call to Action Sharing Unsocial buyer Sharing rate Returning rate Customer Lifetime Value Open rate, engagement Transaction size Emphasis on maximizing cart value, minimizing acquisition costs Bounced Not interested Abandoned Bounce rate Unsatisfied Ratings, delivery issues Feature usage, product discovery
  69. 69. CASE STUDY A A/B testing what really matters B
  70. 70. CASE STUDY B ! 41% increase in revenue per customer! (People bought a lot more product.) ! Conversion also went up, but was secondary in importance.
  71. 71. All business models have issues CAC vs. LTV -- margins are usually very small. A $10M e-commerce business is small. Freemium requires tens of millions of free users. They can be expensive to support. Will enough convert? The average # of apps downloaded by North Americans per month is now 0. Monetizing is incredibly hard. Popularity is fleeting. Chicken & egg problem. Supply and demand. How do you build up both enough? Real monetization requires hundreds of millions of engaged visitors. People’s attention is hard to capture and keep. Content creation. Will it be good enough? Will enough people do it? Why? E-Commerce SaaS Mobile Apps 2-sided Marketplace Media UCG @byosko
  72. 72. You know what business you’re in. You know what stage you’re at. NOW WHAT?
  73. 73. The One Metric That Matters The business you’re in E-Com SaaS Mobile 2-Sided Media UCG Thestageyou’reat Empathy Stickiness Virality Revenue Scale THE ONE METRIC THAT MATTERS @byosko
  74. 74. What really matters when you’re backing up your car?
  75. 75. Moz cuts down on metrics to track SaaS-based SEO toolkit in the Scale stage. Focused on net adds. Net adds up: Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable? Net adds flat: Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support? Net adds down: Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?
  76. 76. Timehop only cares about virality ! Focused on % of daily active users that share content ! Aiming for 20-30% of daily active users to share content “All that matters now is virality. Everything else--be it press, publicity stunts or something else--is like pushing a rock up a mountain: it will never scale. But being viral will.” -- Jonathan Wegener, founder
  77. 77. # of transactions (for merchants) # of nights booked sales total time reading monthly active users monthly recurring revenue (MRR) Examples of OMTM @byosko
  79. 79. Better:
  80. 80. The Layer Cake of Metrics Project OMTM Project OMTM Project OMTM Project OMTM Project OMTM Project OMTM Department OMTM Department OMTM Department OMTM OMTM: Business Help Indicator
  81. 81. What’s your OMTM? ● So what’s your OMTM? Do you know? Can you write it down? Is it available to everyone at your company? ● Can you see how your work matters to the overall health of the business and how you might measure that value creation? @byosko
  82. 82. Drawing lines in the sand
  83. 83. Growth 5% / week (revenue or active users) Time on site 17 minutes Free to paid 2% of free users Mobile file size < 50MB Engaged visitors 30% monthly users 10% daily users Paid load time < 5 seconds Churn 2% / month CLV:CAC 3:1 Some benchmarks @byosko
  84. 84. CASE STUDY: Solare draws a line in the sand @byosko
  85. 85. 50 reservations by 5pm 250 covers that night = CASE STUDY: Solare discovers a leading indicator @byosko
  86. 86. The Lean Analytics Cycle
  87. 87. Identify a key business problem, pick the OMTM, draw a line in the sand, and get started.
  88. 88. Draw a new line ZxLERATOR | NYC | SUMMER 2016 89 LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics Pivot or
 give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data:
 find a commonality Without data: make a good guess Find a potential improvement Draw a linePick a OMTM Lean Analytics Cycle
  89. 89. Quick summary on the Lean Analytics framework ● What you track depends on what type of innovation you’re doing: core, adjacent or disruptive ● What you track depends on your business model and stage (for a startup, project, product or even at a feature-level) ● Find the One Metric That Matters so you can focus as much as possible ● The more holistically you can assess your business, the better off you’ll be (map it all and find the hot spots!) @byosko
  90. 90. The value of data in building better products.
  91. 91. Data is a key input and filter in building better products.
  94. 94. Product & Design (defining goals / objectives) User & customer feedback Sales Marketing Customer Support Etc. ! In-person interviews ! Surveys ! Customer support inquiries ! Real-time online Supported by data Your gut Company vision Collecting Input & Customer Discovery Your own ideas
  95. 95. Data is also a communication tool.
  96. 96. @byosko @byosko
  97. 97. Data is complex. How we communicate it doesn’t have to be.
  98. 98. Alistair Croll @acroll Ben Yoskovitz @byosko