Talk on how to use analytics to optimize your product that I gave at Yelp on February 5, 2014.

1. Lean
Product
Analytics
Dan Olsen
Olsen Solutions
Feb 5, 2014

2. My
Background
n
Educa7on
n
n
n
n
n
20
years
of
Product
Management
Experience
n
n
n
n
n
BS,
Electrical
Engineering,
Northwestern
MS,
Industrial
Engineering,
Virginia
Tech
MBA,
Stanford
Web
development
and
UI
design
Managed
submarine
design
for
5
years
5
years
at
Intuit,
led
Quicken
Product
Management
Led
Product
Management
at
Friendster
CEO
&
Cofounder
of
YourVersion,
“Pandora
for
your
news”
Consultant:
Box,
YouSendIt,
Chartboost,
One
Medical
Will
post
slides
at
hUp://slideshare.net/dan_o
Copyright
©
2014
Olsen
Solu7ons

3. What
does
“Lean”
mean?
n
Lean
Startup
n Achieving
product-­‐market
fit
n Tes7ng
hypotheses
&
learning
n Valida7ng
MVP
with
users
n Improving
&
itera7ng
your
product
quickly
n Minimizing
waste
=
using
resources
effec7vely
Copyright
©
2014
Olsen
Solu7ons

4. What’s
the
Formula
for
Product-­‐Market
Fit?
n
A
product
that:
n Meets
customers’
needs
n Is
beUer
than
other
alterna7ves
n Is
easy
to
use
n Has
a
good
value/price
Copyright
©
2014
Olsen
Solu7ons

5. Dan’s
Model
for
the
Causality
Underlying
Product-­‐Market
Fit
Customer
has
needs
Target
Customer
Customer
Needs
You
design
&
build
product
to
meet
needs
Customer
decides
how
well
product
meets
needs
(sa7sfac7on)
Product
Copyright
©
2014
Olsen
Solu7ons

6. What
are
Customers
Reac7ng
To
When
They
Use
Your
Product?
Feature
Set
UX
Design
Messaging
Copyright
©
2014
Olsen
Solu7ons

7. Valida7ng
New
vs.
Exis7ng
Products
New
Product
Qualita7ve
interviews
Oprah
Exis0ng
Product
Quan7ta7ve
data
Spock

8. +*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
How to be a
Lean Product
Ninja
slideshare.net/dan_o/
+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+

9. Copyright
©
2014
Olsen
Solu7ons

10. Lean
Product
Analy7cs
Process
Iden7fy
What
Your
Metrics
Are
Iden7fy
highest
ROI
idea
Measure
Metrics
Baseline
Values
Evaluate
Metrics
Upside
Poten7al
Global
Level
Select
Top
Metric
Brainstorm
Ideas
to
Improve
Metric
Metric
Level
Learn
&
Iterate
Design
and
Implement
Analyze
How
the
Metric
Changes
Copyright
©
2014
Olsen
Solu7ons

11. Valida7ng
Product-­‐Market
Fit:
Surveys
n
Net
Promoter
Score
Key
follow-­‐up
ques7ons:
• Why
did
you
give
the
score
you
did?
• What
do
we
need
to
do
to
improve?
Copyright
©
2014
Olsen
Solu7ons

12. Qualita7ve
Compliments
Quan7ta7ve
Quant
What?
Qual
Why?
Copyright
©
2014
Olsen
Solu7ons

13. Valida7ng
Product-­‐Market
Fit:
Surveys
n
Survey.io
/
Qualaroo.com
n
“How
would
you
feel
if
you
could
no
longer
use
Product
X?”
n Very
disappointed
n Somewhat
disappointed
n Not
disappointed
n
General
guideline:
40%
or
more
“very
disappointed”
=
product-­‐market
fit
Copyright
©
2014
Olsen
Solu7ons

14. Product-­‐Market
Fit:
Actual
User
Behavior
Trumps
Opinions
n
Asking
a
user
ques7ons
in
an
interview
or
survey
n
n
n
n
Observing
behavior
n
n
n
Valuable,
but…
They’re
telling
you
what
they
think
they
would
do
Measurement
bias
(because
you’re
with
them)
See
what
users
actually
do
Without
you
there
Behavioral
metrics
for
Product-­‐Market
Fit:
n
n
n
n
n
Prospects
sign
up
=
High
conversion
rate
They
keep
using
it
=
High
reten7on
rate
They
use
it
omen
=
High
frequency
of
use
They’re
deeply
engaged
with
it
=
Long
session
7mes
They
pay
for
it
=
Revenue
per
customer
Copyright
©
2014
Olsen
Solu7ons

15. Valuable
to
Have
a
Holis7c
Analy7cs
Framework
Dave
McClure’s
“Startup
Metrics
for
Pirates”
A
A
R
R
R
Focus
on
right
metric
at
right
7me

16. Using
Analy7cs
for
Op7miza7on
n In
addi7on
to
Product-­‐Market
Fit,
you
can
apply
the
Lean
Product
Analy7cs
Process
to
op7mize:
n Your
Business
Results
n Your
User
Experience
Copyright
©
2014
Olsen
Solu7ons

17. Define
the
Equa7on
of
your
Business
“Peeling
the
Onion”
Adver7sing
Business
Model:
Profit
=
Revenue
-­‐
Cost
nique
Visitors
x
Ad
Revenue
per
Visitor
U
mpressions/Visitor
x
Effec7ve
CPM
/
1000
I
isits/Visitor
x
Pageviews/Visit
x
Impressions/PV
V
ew
Visitors
+
Returning
Visitors
N
nvited
Visitors
+
Uninvited
Visitors
I
of
Users
Sending
Invites
x
Invites
Sent/User
x
Invite
Conversion
Rate
#
Copyright
©
2014
Olsen
Solu7ons

18. Equa7on
of
your
Business:
Subscrip7on
Business
Model
Profit
=
Revenue
-­‐
Cost
Paying
Users
x
Revenue
per
Paying
User
N
ew
Paying
Users
+
Repeat
Paying
Users
rial
Users
x
C
onv
Rate
Previous
Paying
Users
x
(
1
–
Cancella7on
Rate
)
T
(
SEO
Visitors
+
SEM
Visitors
+
Viral
Visitors
)
x
Trial
Conversion
Rate
Copyright
©
2014
Olsen
Solu7ons

19. How
to
Track
Your
Metrics
n
Track
each
metric
as
daily
7me
series
Date
Unique
Visitors
Page
views
Ad
New
User
Revenue
Sign-­‐ups
4/24/08
10,100
29,600
25
490
4/25/08
10,500
27,100
24
…
480
…
n
Create
ra7os
from
primary
metrics:
X
/
Y
n Example:
How
good
is
your
registra7on
page?
n Okay:
n BeUer:
#
of
registered
users
per
day
registra7on
conversion
rate
=
#
registered
users
/
#
uniques
to
reg
page
Copyright
©
2014
Olsen
Solu7ons

20. Registra7on
Page
Conversion
Rate
Daily Signup Page Yield Rate vs.
Registration Page Conversionvs. Time Time
New Registered Users divided by Unique Visitors to Signup Page
Daily Signup Page Yield
Registration Page Conversion Rate
100%
90%
80%
70%
60%
50%
40%
Started requiring
registration
30%
20%
10%
Changed
messaging
Added questions
to signup page
0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
Copyright
©
2014
Olsen
Solu7ons

22. View
Each
Business
Metric
as
a
Gauge
Current
Value
Minimum
Possible
Value
Maximum
Possible
Value
Copyright
©
2014
Olsen
Solu7ons

23. Return
(Value
Created)
Priori7zing
Product
Ideas
by
ROI
4
?
Idea
D
3
Idea
A
Idea
B
2
Idea
C
1
Idea
F
1
2
3
4
Investment
(developer-­‐weeks)
Copyright
©
2014
Olsen
Solu7ons

24. Iden7fying
the
“Cri7cal
Few”
Metrics
n
What
is
the
upside
poten7al
of
each
metric?
How
many
resources
will
it
take
to
“move
the
needle”?
n
n
n
Developer-­‐days,
7me,
money
How
much
will
the
needle
move?
Revenue
impact?
Which
metrics
have
highest
ROI
opportuni7es?
Metric
B
Bad
ROI
Return
Return
Metric
A
Good
ROI
Investment
Metric
C
Great
ROI
Return
n
Investment
Investment
Copyright
©
2014
Olsen
Solu7ons

25. Case
Study
from
Intuit
q
q
Improving
UX
Improving
Business
Results
-­‐>
Sign-­‐up
Conversion
Rate
Copyright
©
2014
Olsen
Solu7ons

26. Case
Study:
Account
Signup
Process
Redesign
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80%
60%
50%
40%
30%
20%
10%
1/20/03
1/13/03
1/6/03
12/30/02
12/23/02
12/16/02
12/9/02
12/2/02
11/25/02
11/18/02
11/11/02
11/4/02
10/28/02
10/21/02
10/14/02
0%
10/7/02
Abandonment Rate (7 Day Moving Average)
70%
Copyright
©
2014
Olsen
Solu7ons

27. Analyzed
Drop-­‐Off
at
Each
Major
Sec7on
%
of
Users
100%
80%
100%
Focus
on
biggest
drop
62.3%
60%
58.8%
50.9%
40%
34.4%
32.7%
20%
0%
Sign
in
/
Account
Type
Cash
vs.
Registra7on
Margin
5
Partner
Pages
3
Partner
Pages
Copyright
©
2014
Olsen
Solu7ons

28. Analysis
of
Sign
In/Registra7on
Flow
Open
Account
44%
Register
Registra7on
Process
55%
(24%
of
Total)
45%
drop
off
(20%
of
total)
56%
83%
(46%
of
Total)
Sign
in
Forget
Password
17%
drop
off
(10%
of
total)
64%
of
Total
Account
36%
overall
Selec7on
drop
off
for
this
step
30%
(14%
of
Total)
70%
(32%
of
Total)
Change
Password
80%
(26%
of
Total)
20%
drop
off
(6%
of
total)
Copyright
©
2014
Olsen
Solu7ons

29. Redesigned
User
Flow
Improved
Registra7on
Conversion
Rate
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80%
60%
50%
40%
37% improvement
in conversion rate
Released
New Design
30%
20%
10%
1/20/03
1/13/03
1/6/03
12/30/02
12/23/02
12/16/02
12/9/02
12/2/02
11/25/02
11/18/02
11/11/02
11/4/02
10/28/02
10/21/02
10/14/02
0%
10/7/02
Abandonment Rate (7 Day Moving Average)
70%
Copyright
©
2014
Olsen
Solu7ons

30. Case
Study
from
Friendster
q
Improving
Business
Results
-­‐>
Viral
New
User
Growth
Copyright
©
2014
Olsen
Solu7ons

31. Case
Study:
Op7mizing
Friendster’s
Viral
Loop
% of users
sending = 15%
invites
Active
Users
% of users
who are
active
Invites per
sender = 2.3
Invite
Prospective
Users
Invite
click-through rate
Click
Registration
Process
Fail
Succeed
Don’t
Click
Conversion
= 85%
rate
Users
•
Mul7plied
together,
these
metrics
determine
your
viral
ra7o
•
Which
metric
has
highest
ROI
opportunity?
Copyright
©
2014
Olsen
Solu7ons

32. The
Upside
Poten7al
of
a
Metric
?
100%
100%
85%
15%
0
Registra7on
Process
Yield
Max
possible
improvement
0.15
/
0.85
=
18%
2.3
0
%
of
users
sending
invita7ons
0.85
/
0.15
=
570%
0
Avg
#
of
invites
sent
per
sender
?
/
2.3
=
?%
Copyright
©
2014
Olsen
Solu7ons

33. Okay,
so
how
can
we
improve
the
metric?
How
do
we
increase
the
average
number
of
invites
being
sent
out
per
sender?
n For
each
idea:
n
n What’s
the
expected
benefit?
(how
much
will
it
improve
the
metric?)
n What’s
the
expected
cost?
(how
many
engineer-­‐
days
will
it
take?)
n
You
want
to
iden7fy
highest
ROI
idea
Copyright
©
2014
Olsen
Solu7ons

34. Amer
Launching
Address
Book
Importer…
Copyright
©
2014
Olsen
Solu7ons

35. Amer
Launching
Address
Book
Importer…
Copyright
©
2014
Olsen
Solu7ons

36. Amer
Launching
Address
Book
Importer…
Copyright
©
2014
Olsen
Solu7ons

37. If
you
could
only
track
1
metric
to
measure
your
Product-­‐Market
Fit:
Which
metric
would
it
be?
Copyright
©
2014
Olsen
Solu7ons

38. Reten7on
Rate
n Reten7on
rate
tracks
what
%
of
your
customers
are
s7ll
ac7ve
over
7me
~80%
never use
app again
Curve
eventually
flattens out
Copyright
©
2014
Olsen
Solu7ons

39. Cohort
Analysis
Copyright
©
2014
Olsen
Solu7ons

40. Cohort
Analysis:
Data
Copyright
©
2014
Olsen
Solu7ons

41. Improving
Reten7on
Rate
Over
Time=
Increasing
Product-­‐Market
Fit
David
Skok,
Matrix
Partners
hUp://www.forentrepreneurs.com/saas-­‐metrics-­‐2/

42. Alternate
Ways
to
Track
Reten7on
n Having
lots
of
cohort
curves
is
hard
to
read
n Would
be
great
to
have
a
7me
series
metric
=
one
metric
we
can
track
over
7me
n %
Users
Retained
who
signed
up
X
days
ago
n Can
use
single
or
mul7ple
X
(30
&
90
days)
n Another
metric:
Returning
users
n Good
summary
metric:
#
of
users
“locking
in”
n Gives
a
sense
of
scale
(not
a
%)
n Recommend
7-­‐day
average
(can
do
others
too)
Copyright
©
2014
Olsen
Solu7ons

43. Profitability,
anyone?
Two
key
metrics:
• Customer
Life7me
Value
(LTV)
• Customer
Acquisi7on
Cost
(CAC)
You
want:
LTV
–
CAC
>
0

44. Profitability,
anyone?

45. Profitability,
anyone?
Two
key
metrics:
• Customer
Life7me
Value
(LTV)
• Customer
Acquisi7on
Cost
(CAC)
You
want:
LTV
–
CAC
>
0

46. Life7me
Value
(LTV)
n
Life7me
value
of
a
customer
=
how
much
value
your
average
customer
will
generate
n
LTV
=
ARPU
x
Avg
Customer
Life7me
x
Gross
Margin
ARPU
(Avg
Revenue
/
User)
=
Total
Revenue
/
#
of
Users
n Average
Customer
Life7me
n
n How
long
your
average
customer
generates
revenue
n Equals
1
/
churn
rate
(5%
monthly
churn
=
avg
life
20
months)
n
Gross
Margin:
the
%
of
revenues
lem
over
amer
subtrac7ng
the
cost
of
providing
the
product/service
Note:
for
simplicity,
this
LTV
equa7on
ignores
the
“cost
of
capital”
Copyright
©
2014
Olsen
Solu7ons

47. Customer
Acquisi7on
Cost
(CAC)
n CAC
is
the
average
cost
for
you
to
obtain
a
revenue-­‐genera7ng
customer
n So
it
takes
into
account
both
your
cost
of
acquiring
a
prospect
and
your
conversion
rate
for
conver7ng
prospects
to
revenue-­‐
genera7ng
customers
n CAC=Cost
per
Acquisi7on
/
Conversion
Rate
Copyright
©
2014
Olsen
Solu7ons

48. What
You’d
Like
to
See
Over
Time
n
n
LTV
increasing
as
you
improve
your
value
proposi7on,
customer
reten7on,
&
pricing
CAC
decreasing
as
you
op7mize
your
marke7ng:
segments,
channels,
messaging
Copyright
©
2014
Olsen
Solu7ons

49. Ra7o
of
LTV
to
CAC:
Real
data
from
HubSpot
Copyright
©
2014
Olsen
Solu7ons

50. Lean
Product
Analy7cs
Process
Iden7fy
What
Your
Metrics
Are
Iden7fy
highest
ROI
idea
Measure
Metrics
Baseline
Values
Evaluate
Metrics
Upside
Poten7al
Global
Level
Select
Top
Metric
Brainstorm
Ideas
to
Improve
Metric
Metric
Level
Learn
&
Iterate
Design
and
Implement
Analyze
How
the
Metric
Changes
Copyright
©
2014
Olsen
Solu7ons

51. Questions?
olsensolutions.com
linkedin.com/in/danolsen98
@danolsen
Copyright
©
2014
Olsen
Solu7ons