Measure What Matters: Crafting UX Success Metrics [UX Lisbon 2019, workshop]

Say “hi” and 
introduce yourselves.
(These are the folks who will
make your work awesome.)
3 min
Let’s get to know each other.
Hello!

tweet-friendly
@katerutter

How do you
measure UX
now?

I am not an expert on your business.
You are.
 
 
Today’s goal is to help you explore 
ways to measure how well your product
serves real humans... 
so you can use metrics to design  
& improve the products you build.
What to expect...

It’s NOT about
Right vs Wrong.
 
 
It’s about
Weak ideas made Strong.
What to expect...

You may not like the pace/process/
pressure/perspective, etc. etc.
That’s okay.
All I ask is that you 
keep an open mind and try it.
What to expect...

Let’s get warmed up
Pick an idea to use
for the day
Task-a-
Doodle
write it on a
stickynote
A product you work on
An idea for a new product
or
Product • Feature • Interaction • Customer problem
1 min

a mobile app for
delegating tasks
Working parents with
with kids
need to know when
something’s done
Too much to do, not enough time
need to share
tasks on the go
Task-a-doodle 5 3 2017Who are the people?
(the users)
What problems do
they have?
What is a proposed
solution?
Product Snapshot
BE SPECIFIC
2 min

Our work today…
•Choose your focus
•Identify your customer
•Envision a Key Use
•Establish a Key Metric
•Capture the data
•Track over time
•Tell the story
meaningful… metrics…

None of these numbers help us know 
our product is working for our users.
“Over 10,000 downloads!”
“Avg time on site is 22 minutes!”
“+450 new sign-ins this week!”
Nope.

Surveys
Usability
Testing
Interviews
Observation
Field
research
Metrics &
Analytics
Quantitative
tells you what is
happening
Qualitative
tells you why
things happen
Generative
helps you come up
with new ideas
Evaluative
helps you assess or
validate something
Big data?
best source
of possible
metrics
NUMBERS STORIESUser
Research

Metrics &
Analytics
• Click tracking
• Analytics packages
• A/B testing
• KPIs / OKRs
• Usage stats
• Conversion

DAU Daily Active Users
MAU Monthly Active Users
CPA Cost per Acquisition
ARPDAU Average Revenue per Daily Active User
Retention How long they stay & how  
frequently they interact
Churn How many users you lose vs. keep
LTV LifeTime Value
NPS Net Promoter Score Sketch via Build Better Products by Laura Klein, Rosenfeld Media
Common metrics

Product
Trafﬁc
Retention

Product
Trafﬁc
Retention
use

Product
Retention
Product Level
Feature Level
Interaction Level

“ ~ Joshua Porter
Your metrics will be as
unique as your business.

Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack

Who is our user? 
What do they think/
feel/say?
What do they need?
Ask:

Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
using the task scheduler, task sharer and done
notiﬁcations.
in a way that’s friendly, clear, visual, and fast.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle

5 min
Who is this person? 
(name & picture)
What is their story?
What are their 
needs & goals?
Customer Portrait
BE SPECIFIC

What can our
customer do with our
product that they
can’t do without it?
Ask:

using the task scheduler, task sharer and done
notiﬁcations.
in a way that’s friendly, clear, visual, and fast.
Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle

An example from Foodspotting

rapid viz

smile frown
up brows
neutral
down
brows
no brows
add a mouth...
add eyebrows...
sketch via @luxrco; also see @austinkleon
Expressions Matrix

You can draw a person in 4 simple steps....
Star People

Speed-sketch 
5 star people!
1 min

http://bit.ly/sketchingpeople
Video examples here:

Overhead gallery!

Key Uses 6-up
OK to be obvious
5 3 2017
•1 idea per panel
•6 different ideas
(not a sequence)
•Sketch AND write
What can someone
DO with your
product?
- Solves the problem
- Addresses a need
10 min
Task-a-doodle, Mary

Dot-vote top pick
.5 min
Pick one to focus on: use a colored dot

Key Use
• This is a key use
for your product
• Make the sketch
clean & clear
1 min
Task-a-doodle,
Mary
5 3 2017
Share tasks
with others.
RE-Draw your key use
OK to be obvious

What you have now...
What people?
What problems?
What product?
Product
Snapshot
Who is your
customer? User?
Customer
Portrait
What can a
user do with
our product?
Key
Use
Gives
meaning to
our metrics}

Break!
30 min

…measures the
usage of your 
product by a
person.
a good metric…
Clear &
speciﬁc Comparative
Normalized
Actionable
Changes 
your behavior

• Know the desired user behaviors.
• Know the speciﬁc interactions
that support the action 
or behavior.
• Know what numbers track  
these interactions.
Clear & speciﬁc
share a task
*
a good metric…

Normalized 
(rate or ratio)
• Percentages are
inherently comparative
• They balance out big
changes in total
quantities.
782 total users
420 share a task
1,275 share a task
6,835 total users
18.6%
53.7%
a good metric…

Comparable 
(time-stamped)
• Captured at regular
intervals
• Directly comparable
over a span of time What?!?!
6
a good metric…

Actionable, not vanity
• Total number of users
• Downloads
• Page views
• Likes
• Funding press releases
• Articles in media
• etc…
a good metric…

“ ~ Alistair Croll 
with Ben Yoskovitz
Don’t just ask questions. Know how
the answers to the questions will 
change your behavior.

vanity
total number
of registered
users
unhelpful
sign-ups
% of users
who share a
task 3+  
times a day, 
per week
awesome
better
% of users
who sign in 3+
times a day, 
per week
good
% of new
users per
week
an awesome metric…

So…
How do we get
our awesome
metric?

Brainstorm 3
ideas for metrics
DONE!
What could you count to make sure
the Key Use is happening?

Pick one to focus on
.05 min

3. Is there an object  
basis? 
• “per user” [ ___/user]
• “per user per week” (fancy!)
• Examples: per user, per session,  
or by interaction or item  
(messages per ﬁle, emails  
per thread)
* not required, but super-helpful
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
2. Is there a time basis? 
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...” [ % ]
1. Does the metric  
begin with a number? 
Make it awesome
% of users
who share a
task 3+  
times a day, 
per week
awesome
better
% of users
who sign in
3+ times a
day, 
per week
good
% of new
users per
week
5 min

Share your work
Review the snapshot 
& use.
Peers: make 
recommendations
Improve 
as a pair
Key Metric 2 min 
each

Put it on your dashboard
.5 min

Pick another one
.05 min

3. Is there an object  
basis? 
• Examples: per user, per session,  
or by interaction or item  
(messages per ﬁle, emails  
per thread)
• “per week” [ ___/wk ]
2. Is there a time basis? 
• “number of...” [ # ]
• “percent of...” [ % ]
1. Does the metric  
begin with a number? 
Make it awesome
% of users
who share a
task 3+  
times a day, 
per week
awesome
better
% of users
who sign in
3+ times a
day, 
per week
good
% of new
users per
week
3 min

Key Metric 2 min 
each
Share your work
Review the snapshot 
& use.
Peers: make 
recommendations
Improve 
as a pair

.5 min
Put it on your dashboard

Homework
Pick the third one

.5 min
Pick ONE to work on next

Questions &
Discussion

So…
now
what?

Three ways to gather data: 
• Non-technical 
• Technical “by hand” 
• Instrumented with software
The process of setting up the technology to capture
the numbers that matter to you. This often means
using an analytics package or writing custom code.
Instrumentation

http://www.businessofapps.com/big-list-app-analytics-tools/
https://blog.bufferapp.com/social-media-analytics-tools
Instrumentation

UI Sketch

1.Sketch the UI -
simple is good!
2.Annotate interactions
to capture
3.Make notes on how
to calculate.
Where in the UI
can you capture
data?
UI Sketch
Homework

How do we
track progress
over time?
Ask:

•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?
Metrics Plan

•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?
Metrics Plan
Homework

Track over time

What you have now...
measures
meaningful
metrics}Metric
Tracker
How will we
track progress?
Metrics UI
Sketch
Where is it in
the UI?
Key Metric
Plan
How will we get
the data?
Metrics
Dashboard
What metric
matters?

Tell the story

4 Metrics
Dashboard
Put in this order:
1 Product
Snapshot
2 Customer
Portrait
3 Key
Use
5 Metrics UI
Sketch

Geek out & learn more

Kate Rutter
@katerutter
http://intelleto.com
Thanks!
Swearing, drinking & UX…give it a listen!
http://bit.ly/whatiswrongwithux

Handouts &
Templates

UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Handouts
Crafting UX Success Metrics
Measure what Matters

Features
Users
Uses
Product
Needs/Goals
Company
Purpose
sketches,
prototypes,
pixels
UI
UX Stack
* Via Luxr, Janice Fraser

Product Snapshot
people
problem solution
have
addresses
use
Product name date
/ /

Their story...
Meet
Customer Portrait date
/ /
Needs & goals...

With _________________ , _________________ can...
Key Uses 6-up sketches
date
/ /

With
can...
,
Key Use date
/ /

Feature Set date
/ /
To meet these needs, we have these features:
feature
Notes
feature
Notes
feature
Notes

a great metric...
...makes you look at all the other metrics
and say “none of those other numbers
matter if we don’t get this right ﬁrst.”
a good metric...
...measures the usage of the product by
a person. The usage should be speciﬁc to
features that deliver value to the user.
Making metrics actionable
vanity good betterunhelpful
total number
of registered
users
% of users
who share a
task 3+  
times a day, 
per week
sign-ups
awesome
% of users
who sign in 3+
times a day, 
per week
% of new
users per
week
Does the metric begin with
a number?
• “percent of...” [ % ]
• “number of...” [ # ]
Is there a time basis?
Is it comparative?
• “per week” [ ___/wk ]
Is there an object basis? 
1
Metrics Check
2 3

metric
by
Goal
/ /
Notes
Metrics Dashboard date
/ /
To assess progress, we will measure these metrics:
metric
by
Goal
/ /
Notes
metric
by
Goal
/ /
Notes

Metrics UI Sketch
date
/ /
Here’s what we’ll collect and measure.

by
Goal
/ /
Key Metric plan date
/ /
To assess progress, we
will measure this metric:
Measured
Baseline
Daily Weekly Monthly
Here’s how we’ll collect the data:
metric

Metric Tracker
start date
/ /
by
Goal
/ /
Acquisition
Monthly / weekly
/ /
/ /
/ /
/ /
metric
to
end date
/ /
as of
Baseline
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
Activation
Retention
Referral
Revenue

Metric Tracker

What behavior are you trying to introduce / increase / decrease / change?
Analytics Strategy date
/ /
What is the ﬁnancial impact of the behavior?
Who owns the ﬁnancials for this behavior? Who else is involved?

Measure What Matters: Crafting UX Success Metrics [UX Lisbon 2019, workshop]

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Measure What Matters: Crafting UX Success Metrics [UX Lisbon 2019, workshop]

Similar to Measure What Matters: Crafting UX Success Metrics [UX Lisbon 2019, workshop] (20)

More from Kate Rutter

More from Kate Rutter (20)

Recently uploaded

Recently uploaded (20)

Measure What Matters: Crafting UX Success Metrics [UX Lisbon 2019, workshop]