The document describes a digital diary study conducted over 8 months with 40 people who use fitness trackers. Participants were asked to report on 5 things they do with their tracker each day by submitting a short video and answering questions. The study found that while people believed they checked their trackers frequently, they actually checked them and had the tracker with them less often than assumed. It also uncovered significant technical errors unknown previously to the client. Digital diary studies are effective at gathering contextual information cost-effectively, capturing repeated behaviors, providing instant feedback, and uncovering latent needs.
3. 3
Interviews don’t capture the range of ways people
do things or the rich, in-the-moment details that
people quickly forget.
4. Fitness Tracker Case Study
40 people who use fitness trackers were recruited to participate in
several digital diary studies over 8 months.
The first diary study had them report on 5 things they do with their
tracker. With every entry, they submitted a short video and answered
a few questions.
44
5. 5
Erica, 33
“It’s 10:53pm, and I’m
only at 7388
steps….we’re about to
go take the dog for a
walk. Got to make
sure we get to our
10,000 steps before
the end of the night.”
Show us how you use your fitness tracker
6. Digital diary studies excel at:
• Gathering contextual information cost-effectively
• Capture repeated behaviors in context
• Providing instant feedback on products
• Uncovering latent needs in general problem space
7. What This Presentation Will Cover
What a Diary Study is
Advantages of Diary Studies
Platforms for Conducting Digital Diary Studies
How to Conduct a Digital Diary Study
Highlights for Ensuring an Effective Digital Diary Study
7
9. A Diary Study is a Cultural Probe
Cultural probes use a collection of tools, artifacts
and tasks intended to provoke the user to think
about their environment in new ways….
With minimal intrusion, researchers can glean
insights into participants’ environments that
can help to identify problems and uncover new
opportunities.
9
Adam Starkman & Jennifer Chow
http://designresearchtechniques.com/casestudies/cultural-probes/
10. Key Elements of Diary Studies
Theme-based: the subject and questions are suggested by the
researcher.
Self-reported: the actual reporting is done by the participant, using
whatever platform/prompts were provided.
In-the-moment: the participant is asked to do the reporting
immediately after engaging in the activity, in order to capture as much
detail as possible.
Longitudinal: the goal is to capture a range of uses, rather than a
singular experience. This can include how the user’s experience
evolves over time.
10
11. Traditional Diary Study
Mozilla used paper diary studies in 2012 to learn how participants
were using bookmarks in Firefox.
They asked 10 people to record their experiences with bookmarks
over 3 days.
11
Brian Groudan, Mozilla
research, 2012 /
https://blog.mozilla.org/
ux/page/24/
12. Traditional Diary Study
People came in for an in-person interview afterwards, where they
discussed their entries and sorted them by theme.
Guiding design principles were culled from the analysis that were the
basis for design better bookmarking tools.
12
14. What’s so great about
‘in-the-moment’ feedback?
Can’t you get the same information
in an interview a few days later?
14
15. 15
Most see it like a video camera….
How Does Memory Work?
but really it’s like a slow leak
16. The Psychology of Habits
16The Power of Habit, Duhigg
Behavior trigger
Sequence of actions
occurs repeatedly
Reward / Finish
Our brains turn the
sequence into one
automatic routine;
“chunked”
17. The Psychology of Habits
17The Power of Habit, Duhigg
We can’t access the details of habitual actions until we start them
Diary studies ask people to report when details are still accessible
This also allows for collecting workarounds people are not aware of
Behavior
trigger
Details of
actions become
accessible
Reward /
Finish
Habit
sequence
executed
19. 19
Getting people to actually do the reporting is a
major obstacle for traditional diary studies.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
20. 20
Smartphones are ideal for digital diary studies since
people use them in short spurts throughout the day.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
Average use: 85 times a day*
*Study of 23 people aged 18-33: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0139004
21. Tools for Executing Digital Diary Studies
Platform (web/app)
SMS/Text
Email
Blog
Video
21
22. Digital Diary Study Platforms
22
Name Platform Cost
20/20 Research Web/app diary study high
24Tru Web/app diary study high
CrowdTap Web/app diary study high
UX 360 Web diary study high
Dscout App diary study medium
Dedoose Web diary study low
Typeform online survey low
23. 23
Diary Study
Platform
Advantages:
• Offers analysis; faster to insights
• Lots of data collection options
• Many have panels for recruiting
• Guidance, support, education
• Team members can participate
• Many participants = quant data
Disadvantages:
• Cost
• New tool
• For more tech-savvy participants
• Can’t adjust once launched
$595 for recruiting
$295 for each study
25. Example Dscout Diary Study:
New Product Launch
Goal: Get feedback on a new digital entertainment product
Details: 5 families, 4 months, in-person interviews throughout
Platform: Dscout app, asking them to report every time they used the
service, answering a series of questions.
Key Finding: The service was most robust for mobile viewing, but the
families did the bulk of their viewing on TVs.
Surprising Finding: The study uncovered significant technical errors
that were unknown to our client.
25
26. 26
SMS/Text
Advantages:
• Free
• Good for less tech savvy
participants
• Can adjust on the fly
• More likely to get immediate
response
Disadvantages:
• All analysis is manual
• Limited data input
27. Example SMS Diary Study:
Tracking Diabetes Patients
Goal: Learn how often Type 2 Diabetes patients actually check their
glucose level, as well as how often they have their glucose monitor
with them
Details: 6 participants, 4 weeks, in-person interviews at the beginning
and the end
Daily Text: How are you feeling today? Have you checked your blood
glucose today? Where is your meter now?
Key Finding: People check their glucose & have meter less often than
was assumed. Several aspects were identified that dictate how often
people check their glucose
Study Insight: Asking how they were feeling made them feel cared
for, rather than reminded/judged
27
28. 28
Email
Advantages:
• Free
• Good for less tech savvy
participants
• Can adjust on the fly
• Likely to get more details
than if using SMS
Disadvantages:
• All analysis is manual
• Emails are easy to overlook
29. Example Email Diary Study:
New Product Launch
Goal: Understand how and why people use different channels when
managing their investment account
Details: 18 participants, 2 weeks, in-person interviews at the
beginning and the end
Daily Emails: Questions to understand any activity taken regarding
investment accounts (activity taken, channel used, how felt about it,
etc), plus occasional tasks for probing deeper on specific aspects
Surprising insight: People sometimes conflate how often they think
about their money with how often they check their accounts
Biggest Challenge: Not having enough time to fully analyze data
29
33. The 3 Phases of a Diary Studies
33
The goal:
a concise, focused study
with the right participants.
The goal:
high participation and high
quality data.
The goal:
to turn data into
relevant insights.
34. • Clarify your research question
• Refine and pretest your study
• Minimize the work for your participants
• Recruit the right participants
35.
36. Clarify Your Goals & Research Question
Before Building the Study
Define the goals for the study. What do you want to learn?
Turn that into a research question that will guide your efforts as you
create your diary study.
36
37. Fitness Tracker Study Research Question:
How are activity trackers used by people
attempting to make concrete
improvements to their fitness habits?
37
38. What Type of Diary Study Will
You Build?
Ideation: Collect ideas/suggestions.
“Show me your ideal…”
Inventory: Understand key things participants use, own or do.
“Show me all of your...”
Moments: Capture experiences as they happen in real time.
“Show me every time you…”
Process: Understand key steps of a process in real time.
“Show me your process for…”
Reflection: Explore likes, dislikes, memories and motivators.
“Tell me how you feel about….”
38
http://dscout.helpscoutdocs.com/article/121-tutorial-2-building-your-mission
39. 39
Consider how often you ask them to report. It
should be enough to collect useful data, but not
enough to cause fatigue.
Help Your Participants Succeed By
Reducing the Reporting Burden
40. 40
Pairing themed multiple-choice questions with
open-ended ones creates a logical flow for people
and answers both “what” and “why”.
41. 41
Another benefit of multiple-choice is you can
quickly see trends as the data comes in.
42. 42
Open-ended questions provide the richest data but
are more work for both for you and the participant.
Use them sparingly.
I think it should send reminders on when to be more
active when I'm lagging in my goal such as vibrating.
I do wish that other activities
could be tracked with the
Fitbit without having to input it
myself.
Maybe if I could get
some sort of notification
on my phone if a specific
friend or group of friends
had passed me.
43. 43
Pretest by sitting down with someone as they go
through every step. Look for awkward or unclear
phrasing.
44.
45. 45
Diary studies generate a lot of data.
It is good to start small.
10 x 10 x 10 =questions participants entries
data points
1000
46. 46
If recruiting through Dscout, you’ll only pay
incentives for participants that finish. Expect to
pay $5-$20 per entry.
We paid $25 for a
total of 6 entries.
If you’re working with a
specialized or busy
population, you may
need to pay as much as
$20/entry.
47. 47
We looked for participants who have a fitness
tracker and were regularly using it as part of a
fitness or weight loss goal.
Find the Right Participants
48. • Review & make notes from your data
as it’s coming in
• Be persistent with participants
• Ask them questions for clarity
• Do post-study interviews
49.
50. Watch the data as it comes in.
This is the fun part.
Are your questions holding up?
If not, can you fix it?
Either way, learn for next time.
50
51. 51
Why did you get your first fitness tracker? Larry, 31
“I got a fitness tracker because
was about 300 lbs, on a diet,
exercising, wasn't losing weight. I
knew there was something
missing. I wanted to learn how
many calories I was burning so
could know how little or much I
should be eating. That was the
missing link. I have lost 150 lbs,
and the only way did it was
because I started using Jawbone
Up.”
52. 52
Take notes of interesting patterns and questions
to ask participants as you go through it.
53.
54. 54
Minimize your drop off rate. Stay in touch with
your participants. Be friendly and persistent.
55. 55
Ask them questions if you want more clarity about
their answers. Be appreciative and interested in
what they have to say.
56. 56
Do follow-up interviews with the most engaged
participants. You already have rapport and can dig
deeper into their reporting and actual behaviors.
57. • Use the built-in analysis tools
• Start by answering your initial question(s)
• Identify patterns and/or exceptions
• Share your findings with real examples
58. 58
The good news is that by simply reviewing and
making notes on the data, you have already
started on the analysis.
59. 59
Diary Study platforms offer tools to analyze the
multiple choice and rating questions, reducing
the work the researcher has to do.
60. 60
If analyzing manually, data should be entered into
a spreadsheet so it can be coded, refined and
sorted.
61. What Further Analysis Should be Done?
That depends on many variables, such as:
• What your initial question/s were
• How surprised you are by the data that has come in
• How it fits into other research methods
• Your final deliverables
61
63. Main Benefits of Using Diary Studies
• To collect rich stories about how and why people do things
• Get intimate glimpses of how a product fits into people’s lives
• Allows for understanding evolving use of a product
• Uncover latent needs for new product development
• Cost-effective way to collect contextual insights
63
64. Checklist for a Great Diary Studies
Phase 1, Prepare:
• Have a clear, focused research question to guide your efforts
• Have an appropriate mix of closed/open-ended questions
• Pre-testing it to ensure clarity and understanding
• Start small & recruit the right participants
• Reduce the burden on them
Phase 2, Execute:
• Watch the data and follow up with participants during the study
• Do follow-up interviews with the best participants
Phase 3, Analyze:
• Answer your initial question first
• Give yourself enough time to analyze the data
64
65. Tips for Building Your Own Diary Study
Before you can use it, you may have to sell it. Here are a few things to
keep in mind:
• It’s an extremely flexible tool; make it work for your needs
• Start small, let it prove it’s value to your organization and yourself
(no-cost platform, 5-6 participants)
• Evangelize the insights and stories you collect to build more
support.
• You become an expert by doing. So dive in!
65
66. For a 4-min video about diary studies:
www.gotoresearch.com/speaking/sara/
Questions about diary studies? Ask!
sara@gotoresearch.com
@ maybanks