Metrics: Making Your Product Count by Dan Corbin and Wes Kay
•Download as PPTX, PDF•
4 likes•749 views
Building the right thing at the right time is sometimes hard to do. Even when you feel like your product has hit product/market fit or is scaling beautifully, how do you know you picked indicators that lead to innovation and/or scale? Actionable metrics and qualitative analysis that aren't based on traditional vanity metrics can help. In this round of Product SIG, Wes and Dan will start to shine a light on a few KPI-driven approaches and the value-adds and problems that can come with each. Wes Kay Wes is both an Agile/Lean practitioner and coach, with a passion for both product and process. He has traveled the globe to certify with and learn from Agile luminaries such as Mike Cohn, Ilan Goldstein, and Colin Tan (the authors of Scrum Shortcuts), and he has coached engineering and product teams at Dell, Humanity, ShiftPlanning, and TeacherGraph. Dan Corbin Dan is the API Product Manager for Context.IO (a Return Path company). He is a Certified Scrum Product Owner and a Certified Scrum Professional with over 15 years of product development, project management, and programming experience
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (13)
Recently uploaded
Recently uploaded (20)
Metrics: Making Your Product Count by Dan Corbin and Wes Kay
- 1. Metrics: Making Your Product Count Agile Austin - Product SIG February 24, 2015
- 4. Wes Kay Agile Coach | Lean Product Owner (Macmillan New Ventures, Dell, Humanity, ShiftPlanning, and TeacherGraph) Dan Corbin API Product Manager, Context.IO (a Return Path company)
- 6. What are Metrics, and why do we use them?
- 7. What is a Key Performance Indicator (KPI)?
- 8. What Are Vanity Metrics?
- 9. “Vanity metrics are dangerous.”
- 10. Vanity Metrics = Metrics w/o Actionable Meaning
- 12. Challenge Your Assumptions (In)validate Those Assumptions Measure What Matters
- 13. Actionable Metrics: Macro vs. Micro
- 14. Macro Metrics Baselining, Learning, and KPI’s
- 18. Micro Metrics Optimization and Troubleshooting
- 19. Hill Climbing
- 23. Your Metrics Will Change
- 24. What metrics should you track?
- 26. “Not sure what the numbers mean? Get the customers on the phone and ask them.” -Eric Ries
- 28. Recommended Sites and Blogs PracticeTrumpsTheory.com LeanStartupCircle.com SVProduct.com
- 29. More questions? How do you utilize metrics? Thank you for joining us
- 30. Contact Us dan.corbin [at] gmail.com me [at] weskay.com
- 31. Tools and Resources Optimize.lyGoogle Analytics
- 32. Tools and Resources KISSmetrics Crazy Egg
- 33. Tools and Resources Geckoboard Moz
Editor's Notes
- Data driven decisions are critical if you want a successful product or business. Being “data driven” is easier said than done. You need to ask the right questions and determine which data to use and which you can ignore. This is difficult because relying on activity and lagging metrics is like driving while looking in the rearview mirror. You’ll need to learn which metrics are the true indicators of your product’s potential and future. Demystifying “gut” feelings. Creating a shared base of understanding/knowledge.
- A KPI is a measurable value that demonstrates how effectively we achieve critical business objectives. Companies and organizations often use KPIs to evaluate their success at reaching targets. Your KPIs will be different from other companies and even from people within your companies. The key is standing these metrics against your business’ objectives.
- Vanity metrics are the opposite of actionable metrics. They may document the current state of the product but offer little to no insight into how we got here or what to do next. Basically, these are numbers or stats that look good on paper, but don’t really mean anything important. Examples of vanity metrics include web hits, visitors, number of downloads, subscribers, Gross Site Visits, Page Views, Gross Revenue. They aren’t completely meaningless, but they aren’t what you should be spending the majority of your time or energy on. Vanity metrics can’t tell us anything about the levers that affect scale or innovation. The levers behind them are too obfuscated to inform accurate decision making on reasonable scale. Data that doesn’t inform our decision making?
- An actionable metric is one that ties specific and repeatable actions to observed results. These results line up with the business goals you have defined. If a metric isn’t actionable and you can’t do anything to make it better then why are you tracking it? What are examples of Actionable Metrics? Early on Facebook had 150K users. Considering there was bigger competitors out there (MySpace, Friendster, etc), this number doesn’t tell you much. A better metric was the % of users that visited multiple times a day (75%). Another was the % of students using Facebook one month after it launched on a new campus (90%!)
- Experiment early and often. Scientific approach. Measure against your hypothesis. Using metrics to simply and objectify decision making. A/B testing is one way to pinpoint the best changes to make. For example, The open rate of two email subject lines in an A/B test will immediately tell you which subject line to pick.
- Most prevalent example of Macro Metrics in Lean Startup: Pirate Metrics: AARRR (Acquisition, Activation, Revenue, Retention, Referral)
- Dave McClure is an entrepreneur and prominent angel investor based in the San Francisco Bay Area, who founded and runs the business incubator 500 Startups. It's not enough to analyze your data and produce awesome graphs based on your subscribers' behavior. We do that, sure, but our main goal is making specific recommendations that will increase your revenue. Pirate Metrics figures out what your paying customers have in common, and suggests actions you can take to attract similar new users.
- Measuring against like customer segments. Cohort analysis is a subset of behavioral analytics that takes the data from a given eCommerce platform, web application, or online game and rather than looking at all users as one unit, it breaks them into related groups for analysis. These related groups, or cohorts, usually share common characteristics or experiences within a defined timespan. Cohort analysis allows a company to “see patterns clearly across the lifecycle of a customer (or user), rather than slicing across all customers blindly without accounting for the natural cycle that a customer undergoes.”
- Cohort analysis is a subset of behavioral analytics that takes the data from a given eCommerce platform, web application, or online game and rather than looking at all users as one unit, it breaks them into related groups for analysis. These related groups, or cohorts, usually share common characteristics or experiences within a defined time span. Cohort analysis allows a company to “see patterns clearly across the lifecycle of a customer (or user), rather than slicing across all customers blindly without accounting for the natural cycle that a customer undergoes.” Alistair Croll; Benjamin Yoskovitz. Lean Analytics: Use Data to Build a Better Startup Faster. Sebastopol, CA: O'Reilly. ISBN 1449335675.
- Optimizing user behavior, user flows, and parts of complete actions. Don’t lose sight of your overall goals and the macro metrics that hopefully reflect your business’ goals.
- Don’t lose sight of the forest for the trees. Don’t focus so hard on the peak in front of you that you lose sight of the peaks that may surround and be available to you. In computer science, hill climbing is a mathematical optimization technique which belongs to the family of local search. It is an iterative algorithm that starts with an arbitrary solution to a problem, then attempts to find a better solution by incrementally changing a single element of the solution.
- A local optimum of an optimization problem is a solution that is optimal (either maximal or minimal) within a neighboring set of candidate solutions. This is in contrast to a global optimum, which is the optimal solution among all possible solutions, not just those in a particular neighborhood of values. Mitigation concepts: split testing (A/B), qualitative analysis Vary your experiments widely enough to prevent this path.
- Clarity Once you establish metrics, be sure that everyone in the organization understands: what they are how they are to be used how they will be calculated where the data comes from exactly why it is important It’s important to add the KPIs to information radiators (highly visible displays used to track progress).
- Metrics change as a business grows When beginning you may look at web traffic, followers, subscribers, reviews, social media shares, etc. As you grow you’ll want to track thing such as sales, revenue, conversion rate, customer satisfaction, etc. A more mature company will track: profit, retention length, churn rate, revenue per customer, lifetime value, etc.
- Ultimately it’s up to you: You need to first determine the goals of your business. This is critical. Only then can you figure out which metrics you should be tracking. Actionable metrics use the scientific method to pinpoint exactly what change will lead to the outcome you’re trying to create. When creating your metrics, make sure they satisfy the three A’s: Actionable, Audited, and Accessible.
- Building a product without metrics can feel like finding your way through a dark room, no matter how well acquainted you are with your domain.
- Google Analytics - Helps you understand user behavior Optimizely - This lets you quickly make changes to your site and then automatically tests which version—the old version or your new version—is better at accomplishing your goals.
- KISSmetrics - takes general Google Analytics data, which is great, and adds granularity around the people who are using your site. Being able to access that level of detail enables you to test content and design much more effectively. Funnel Metrics A/B tests Revenue Reports by channel Cohort analysis CrazyEgg - Creates an awesome heat map or confetti map so you can visually see what’s happening on your site (tracks mouse, clicks).
- Geckoboard - Helps you gather your key metrics in one place. Use Geckoboard to communicate your data instantly across your organization. Moz - Helps you understand relationships between content, search marketing, social activity, brand mentions and inbound links. Moz provides analytics as well as actionable next steps.