More Related Content
Similar to World-Class Web Metrics by Dan Olsen (20)
World-Class Web Metrics by Dan Olsen
- 2. World‐Class Metrics:
Amazon.com
$37 B Market Cap
$19 B Revenue ‘08
•Automated testing
of page elements &
suggested products
•High degree of
user personalization
•Real‐time
recommendations
Copyright © 2009 YourVersion
- 3. World‐Class Metrics:
Google
$137 B Market Cap
$22 B Revenue ‘08
•Auto‐testing of
UI elements &
search results
•Automatically
learns from user
mistakes
•Optimize ads to
maximize revenue
Copyright © 2009 YourVersion
- 5. My Web Metrics Epiphany
Quicken (1998 – 2001)
New version every year: shrink‐wrapped CD‐ROM
Pre‐launch
Before development: Validated product ideas with users
During development: Usability testing & user feedback
Post‐launch
No “real‐time” metrics data
Sales data: 1‐2 month lag, not actionable
User surveys: actionable, but not until next product cycle
Quicken Brokerage (2001 – 2003)
Online brokerage launched in September 2002
The day after launch: LOTS of metrics data → sought more
Much faster iteration: metrics → hypothesis → improvement
Copyright © 2009 YourVersion
- 6. Metrics Maturity Model
Unaware Aspiring World‐Class
Metrics Comprehensive
None Sporadic
Tracking breadth & depth
Analysis None Ad hoc Recurring
Automated
Testing & Some A/B
None multivariate
Optimization tests
testing
Lack of Partial Clear, dedicated
Organization
ownership ownership owners
Intuition‐ Data‐ Continuous
Culture
based driven improvement
Copyright © 2009 YourVersion
- 8. Approaching Business as an
Optimization Exercise
Given reality as it exists today,
optimize our business results
subject to our resource constraints.
Copyright © 2009 YourVersion
- 9. Define the Equation of your Business
“Peeling the Onion”
Profit = Revenue ‐ Cost
Paying Users x Revenue per Paying User
New Paying Users + Repeat Paying Users
Trial Users x Conv Rate Previous Paying Users x ( 1 – Cancellation Rate )
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate
Copyright © 2009 YourVersion
- 10. How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New User
…
Date Visitors views Revenue Sign‐ups
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Create ratios from primary metrics: X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page
Copyright © 2009 YourVersion
- 11. Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page
100%
90%
80%
Daily Signup Page Yield
70%
60%
50%
40%
30% Started requiring
registration
20%
Changed Added questions
messaging to signup page
10%
0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
Copyright © 2009 YourVersion
- 12. Identifying the “Critical Few” Metrics
What are the metrics for your business?
Where is current value for each metric?
How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?
Metric A Metric B Metric C
Good ROI Bad ROI Great ROI
Return
Return
Return
Investment Investment Investment
Copyright © 2009 YourVersion
- 15. Using Analytics to Clarify The
Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customers’ needs
Is easy to use
Has a good price
Is better than other alternatives
Offline: hard to measure
Online: can measure everything
Measuring value: # of users, repeat use, revenue
Can achieve clarity with metrics and analytics
Copyright © 2009 YourVersion
- 16. Once You’re Measuring Everything,
Where Do You Focus?
How do you decide which user benefits
and features you should focus on?
Need framework to make decisions
“Problem space” vs. “Solution space”
Importance vs. Satisfaction
Importance of user need (problem space)
Satisfaction with how well your product meets
the user’s need (solution space)
Copyright © 2009 YourVersion
- 17. Importance vs. Satisfaction
Ask Users to Rate for Each Feature
100 98
Great
95
84 87
90
Bad 86
85 79 84
55 70
80
Importance
80
75 72
80
70
75
65
60
55
41
50
40 50 60 70 80 90 100
Satisfaction
Recommended reading: “What
Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion
- 18. Kano Model: User Needs & Satisfaction
User Satisfaction
Delighter (wow)
Performance
(more is better)
Need Need
not met fully met
Must Have
Needs & features
migrate over time
User Dissatisfaction Copyright © 2009 YourVersion
- 19. Olsen’s Hierarchy of Web User Needs
(adapted from Maslow)
Customer’s Perspective What does it mean to us?
How easy to use is it? Usability & Design
Satisfaction
Increasing
Does the functionality
Feature Set
meet my needs?
Does the functionality work?
Absence of Bugs
Dissatisfaction
Decreasing
Is the site fast enough?
Page Load Time
Is the site up when I want to use it?
Uptime
Copyright © 2009 YourVersion
- 20. Using Web Analytics Tools to
Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback
Copyright © 2009 YourVersion
- 21. Basic Tracking of Traffic:
Google Analytics
•Unique
visitors
•New vs.
returning
•Pageviews
•Time on site
•Top referrers
•Top geos
Copyright © 2009 YourVersion
- 22. Seeing Where Users are Clicking:
CrazyEgg Heatmap
•Shows Click
Density
•Color indicates
% of clicks
•See which links
perform best
•See impact of
UI changes
Copyright © 2009 YourVersion
- 23. Measuring Key Conversions:
Conversion Funnel
•Tie user actions to
business goals
•Instrument key steps in
user flow
•See where users are
dropping off
•Quantify improvement
from changes
Copyright © 2009 YourVersion
- 24. Monitoring User Feedback
Quantitatively with Kampyle
Unobtrusive
Solicitation Average Grade over time
Simple popup
Average Grade
for each page
on your site
Copyright © 2009 YourVersion
- 25. The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design
Information
Architecture
Conceptual
Design
Recommended reading: Jesse James Garrett’s
“Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion
- 27. Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B
The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold
Copyright © 2009 Olsen Solutions LLC
Copyright © 2009 YourVersion
- 31. Organization and Culture
Impact Product Success
Roles and Skills required
Product Management (PM)
UI Design
Development
Quality Assurance (QA)
Operations
Cultural traits required
Customer‐centric
Data‐driven decision‐making with clear ownership
Rapid, iterative development
Continuous improvement mindset
Copyright © 2009 YourVersion
- 32. The Product Manager’s Job:
A Successful Product
Be the expert on the market and the customer
Translate business objectives and customer
needs into product requirements
Be the clearinghouse for all product ideas
Work with team to design & build great product
Define and track key metrics
Use metrics to drive continuous improvement
Identify, plan & prioritize product ideas to
maximize ROI on engineering resources
Copyright © 2009 YourVersion
- 33. Prioritizing Product Ideas by ROI
?
Return (Value Created)
4
Idea D
3
Idea A Idea B
2
Idea C
1
Idea F
1 2 3 4
Investment (developer‐weeks)
Copyright © 2009 YourVersion
- 34. Case Study 1: Quicken Brokerage
Optimizing Sign In/Registration Flow
Open
Account
55%
44% Register Registration (24% of Total)
Process
45% drop off
64%
(20% of total)
of Total Account
36% overall Selection
83% 30% drop off for
56% (46% of Total) (14% of Total) this step
Sign in
Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)
17% drop off 20% drop off
(10% of total) (6% of total)
Copyright © 2009 YourVersion
- 36. Case Study 2: Friendster
Optimizing Viral Growth
% of users
sending = 15% Invites per Invite
= 20%
invites sender = 2.3 click-through rate
Active Invite Prospective Click Registration Fail
Users Users Process
% of
users who = 50% Succeed
Don’t
are active Click Conversion
rate = 85%
Users
• Multiplied together, these metrics determine your viral ratio
• Which metric offers the highest ROI opportunity?
Copyright © 2009 YourVersion
- 39. Adding Metrics and Optimization to
your Product Process
Site Level
Business Product Prioritized
Plan Objectives Objectives Feature List
Scoping Feature
Level
Requirements
Design & Design
Code Test Launch
Develop
Metrics & User
Optimize Feedback
Copyright © 2009 YourVersion
- 40. Optimization through Iteration:
Continuous Improvement
Measure
the metric
Analyze
Learning the metric
Gaining knowledge:
• Market Identify top
• Customer opportunities
to improve
• Domain
• Usability Design & develop
the enhancement
Launch the
enhancement
Copyright © 2009 YourVersion
- 42. Questions? www.yourversion.com
dan@yourversion.com
Copyright © 2009 YourVersion