In this workshop, you will be introduced to the field of design research through a broad range of design research methods. The workshop will focus on several selected methods that anyone can use, regardless of background and job description. You will learn about selected quantitative methods (usage data, analytics, in-app audits) and qualitative methods (interviews and field immersion), and how to combine quantitative and qualitative methods to fit your project needs. Through this workshop, you will also find out ways to share findings effectively and turn them into actionable projects within the organisation.
2. @arasb #uxsg
User research is asking deliberate questions
about a problem you care about, working
with your users methodologically to find out
answers and needs, discussing findings with
your team to design possible solutions.
3. @arasb #uxsg
Find a problem
and the questions
Work with users
to discover
answers
Analyze & discuss
with the team
29. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
30. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Research procedures are
usually very flexible.
Scientific rigor is not required in many
methods. It is not only OK, but desired to
tailor the procedures to project needs.
31. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Find a problem
and the questions
Work with users to
discover answers
Analyze & discuss
with the team
32. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Define the
problem
Formulate
questions
Plan
inquiry
Carry it
out
Analyse
findings
Share
33. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Define the
problem
Formulate
questions
Plan
inquiry
Carry it
out
Analyse
findings
Share
34. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Define the
problem
Formulate
questions
Plan
inquiry
Carry it
out
Analyse
findings
Share
35. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Formulate
questions
Plan
inquiry
Carry it
out
Analyse
findings
Share
36. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Define the
problem
Formulate
questions
Plan
inquiry
Carry it
out
Analyse
findings
Share
37. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Questions for
the interview
Carry it
out
Analyse
findings
Share
38. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Questions for
the interview
Carry it
out
Analyse findings in great
strategic detail
Share
39. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Research procedures are
usually very flexible.
Scientific rigor is not required in many
methods. It is not only OK, but desired to
tailor the procedures to project needs.
40. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Validity is more important
than statistical significance.
Many methods do not require statistical
significance to produce useful insight.
41. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Quantitative and qualitative
methods can be combined.
One is not better than the other. They
produce different types of insight.
42. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
It is useful before, during,
and after the project.
Generative, descriptive and evaluative
research reveal different insights.
43. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Generative
Descriptive
Evaluative
44. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Planning:
What is the need?
What drives it?
Is this real?
Execution:
Are we meeting the need?
Are we getting close?
Can there be alternatives?
Live product:
Have we missed anything?
What are the new needs?
How do people feel?
45. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Pilots/Betas are too late in
the product cycle.
You may have already built something that
no one cares about.
46.
47.
48.
49. I thought we were going to build a better
Facebook. But within 30 minutes after
the release, I realized, Oh my God, it's
broken. Holy shit, we totally fucked up!
-Bill Nguyen
50. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Pilots/Betas are too late in
the product cycle.
You may have already built something that
no one cares about.
51. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Research accelerates innovation
Serving unmet needs is essential for innovation.
User research finds those needs.
52. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
Your competition is not
taking a vacation.
Benchmarking yourself against others
guarantees that you remain ahead.
53. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
End states give us only
partial information.
Correlation does not mean causation,
“why” is just as important as “what.”
54. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
60. User Tracks
Users leave tracks in data.
Look at these traces to find
potential problems.
@arasb #uxsg
61. @arasb #uxsg
User Content
• Anonymized form fields
• Complaints
• Significant outbound calls
• User comments (app stores,
customer sites, social)
Analytics
• Top entry and exit paths
• Top referrers
• Search keywords
• Bounce rates
• Time on page
• Session length
• Visitor flow/Funnels
Usage Data
• Top features
• Collection sizes
• Time of use
• Frequency of use
• Device distribution
Prioritize HypothesizeQuick wins
62. User Tracks
Users leave tracks in data.
Look at these traces to find
potential problems.
@arasb #uxsg
What isn’t working?
What can we do better?
Day 1: Identify the numbers that describe
user behavior, work with the team
to pull that data.
Day 2: Understand the “what”, create
hypotheses, get more data to see
how strong they are.
69. @arasb #uxsg
Content
• Tone of voice
• Timeliness
• Detail
• Accuracy
• Authenticity
• Content types
• Update frequency
• Reuse
70. Benchmarking
A fact-based snapshot of
how the product/service
compares with others.
@arasb #uxsg
How do we honestly
compare against others?
Day 1: Establish criteria, identify competitors.
Day 2: Divide amongst the team, fill and grade.
Day 3: Fact-checking and cross-ranking.
Use existing data where available.
Ask finance, call center and field support.
72. @arasb #uxsg
Preparation
• Talk with your champion.
• Study the org chart.
• Inquire about organizational
dynamics.
• Find out about recent
successes and failures.
• Pay attention to ego-boosters
and sore spots.
• Distill 10-15 questions, as a
guide for yourself.
Interviews
• Be yourself.
• Ask questions selectively.
• Go in ascending stratified order.
74. @arasb #uxsg
Preparation
• Talk with your champion.
• Study the org chart.
• Inquire about organizational
dynamics.
• Find out about recent
successes and failures.
• Pay attention to ego-boosters
and sore spots.
• Distill 10-15 questions, as a
guide for yourself.
Interviews
• Be yourself.
• Ask questions selectively.
• Go in ascending stratified order.
• Avoid interrogation and sales.
• Avoid masochism, servitude and
cordial hypocrisy.
75. @arasb #uxsg
What is at stake?
Do they know that?
Day 1: Prepare for the interview, chat
with your champion.
Day 2: Carry out your interviews.
Day 3: Analyze and prepare a pithy,
crystal-clear executive summary.
Stakeholder
Interviews
Making sure that you and
the sponsors are hearing
each other correctly.
90. Field guide for beginners
• Dress a bit less casual than normal.
• Pair if possible.
• Think with a beginners’ mind.
• Ask and shut up.
@arasb #uxsg
91. “What do you think of this feature?”
“How would you design this?”
“My name is Albert, I built this.”
“Wow I didn’t realize it was THIS hard!”
@arasb #uxsg
92. Focus on Behavior, Not Attitude
“When was the last time you did X?”
“Can you show me how you normally do X?”
“You mentioned X, can you talk a bit about it?”
@arasb #uxsg
93. Field guide for beginners
• Dress a bit less casual than normal.
• Pair if possible.
• Think with a beginners’ mind.
• Ask and shut up.
• Don’t forget the research question.
• Don’t take notes verbatim, listen first.
@arasb #uxsg
94. @arasb #uxsg
How do customers pay their bills in the branch?
Observation Thoughts Actions
Unsure about
which window
to go to
Show
upcoming
tickets in each
window
95. Field guide for beginners
• Dress a bit less casual than normal.
• Pair if possible.
• Think with a beginners’ mind.
• Ask and shut up.
• Don’t forget the research question.
• Don’t take notes verbatim, listen first.
• Take pictures/video with permission.
@arasb #uxsg
Watch out for
• Desire to help
• Confirmation bias
• Anchoring effect
• Personal agendas
• Organizational dynamics
• Sample selection
96. Field
Immersion
A shocking, fast-acting
calibration on prejudices
and assumptions.
@arasb #uxsg
What are they doing?
Why are they doing it?
Day 1: Identify questions, let your users know
that you are coming for a visit.
Day 2: Visit the site in the morning, complete
and organize notes in the afternoon.
Day 3: Prepare for analysis by going through
your notes and organizing themes.
104. “Ivory Tower” Analysis
• Opaque and arrogant
• Hard to comprehend the underlying
reasons
• Closed to contribution
• Not helping others improve the insights
• If wrong, creates a big loss of time
@arasb #uxsg
Shared Analysis
• Open and collaborative
• Easy to understand and challenge
• If wrong, caught very early
• Team annotations
105. @arasb #uxsg
How do customers pay their bills in the branch?
Observation Thoughts Actions
Unsure about
which window
to go to
Show
upcoming
tickets in each
window
106. “Ivory Tower” Analysis
• Opaque and arrogant
• Hard to comprehend the underlying
reasons
• Closed to contribution
• Not helping others improve the insights
• If wrong, creates a big loss of time
@arasb #uxsg
Shared Analysis
• Open and collaborative
• Easy to understand and challenge
• If wrong, caught very early
• Team annotations
• Focused analysis techniques
108. @arasb #uxsg
SCAMPER
Substitute
Combine
Adapt
Modify
Put to other use
Eliminate
Reverse
Use Facebook, Twitter, Google login
Make it a part of checkout
Change the login prompt
Ask for less info, simplify passwords
Ask for personal info each time
Don’t ask for login, rely on a cookie
Force login, make logging out harder
Visitors are not logging in.
110. @arasb #uxsg
Visitors are not
logging in.
They are not
seeing the link.
Links are low-
contrast b/c
brand guidelines.
Our brand is too
bland and boring.
Why?
111. @arasb #uxsg
Visitors are not
logging in.
They are not
seeing the link.
Links are low-
contrast b/c
brand guidelines.
Our brand is too
bland and boring.
They don’t want
to give us info.
They don’t trust
us.
We share their
info with our
partners.
112. @arasb #uxsg
Visitors are not
logging in.
Show the benefits
of logging in.
Talk about our
loyalty program.
Make a banner
near login form.
How might we?
114. “Ivory Tower” Analysis
• Opaque and arrogant
• Hard to comprehend the underlying
reasons
• Close to contribution
• Not helping others improve the insights
• If wrong, creates a big loss of time
@arasb #uxsg
Shared Analysis
• Open and collaborative
• Easy to understand and challenge
• If wrong, caught very early
• Team annotations
• Focused analysis techniques
• Open discussion and consensus
115. Get everyone involved in
analysis to get a very rich
understanding.
@arasb #uxsg
Shared Analysis
Here’s what we didn’t know.
Here’s what we can do!
Day 1: Take time to work on analysis, explore
your data, call your participants if you
need to clarify points.
Day 2: Meet with the entire team, share your
why/HMW work. Discuss in detail,
identify 5-10 areas to work on as a team.
117. Presentation structure
• Always relate back to the original
research question!
• Start with the problem
• Briefly tell what has been done
• Share 5-8 findings
• Show the research material
• Be transparent about your process
@arasb #uxsg
Make it impactful and compact
• Group recurring themes.
• Items that confirm you and
your team’s assumptions.
• Items that completely destroy
the project.
119. @arasb #uxsg
High Bounce Rates
Very low Q3 Performance
Increase in Phone Inquiries
Question:
How do our customers
in mid and premium
segments learn about
loans on mobile?
121. Loan pages are not inviting
the users to explore.
@arasb #uxsg
122. @arasb #uxsg
“I yawn when I look at these”
Mid-segment, 34 years old, customer since 1997
123. Loan pages are not inviting
the users to explore.
Possible solutions:
1
2
3
4
@arasb #uxsg
124. Presentation structure
• Always relate back to the original
research question!
• Start with the problem
• Briefly tell what has been done
• Share 5-8 findings
• Show the research material
• Be transparent about your process
@arasb #uxsg
Make it impactful and compact
• Group recurring themes.
• Items that confirm you and
your team’s assumptions.
• Items that completely destroy
the project.
• Ask for new audiences!
125. Snowball
Meetings
Share findings openly and
let every team absorb as
much as they can.
@arasb #uxsg
Help us talk to all teams
to improve our product.
Day 1: Review your analysis with the team.
Prepare a rough outline, storyboard
your narrative. Use pen/paper/walls.
Day 2: Create your presentation. Include
the data that lead to the research
questions. Compile a strong set of
quotes, videos, photos from the field.
130. @arasb #uxsg
Aras Bilgen
Summary of our research
Perona Iner (BBVA Compliance Manager)
Hi Perona,
Please find our research report attached. You can view the
prototype showing our suggestions at the following link.
http://prototype.link/dkJd2ao
Regards,
Aras
User research
report 13-11-
2016.docx
[Auto-saved]
133. @arasb #uxsg
Narrative
Prototypes
Infuse solution reviews
with your research process
and insights.
Here’s what we should do,
and here is why.
Day 1: Design a few screen, flow and
content improvements. Work in
high-fidelity, if possible.
Day 2: Add the rationale highlights from
your research findings.
135. Focus groups? Hard to moderate, strong group dynamics,
potential for bias, lack of focus.
Targeting is hard, self-report may
not be accurate.
Surveys?
Recruitment is very hard for a wide user
base, relatively easy for niche services,
enterprise apps.
Interviews?
Getting to the right users is hard, different
motivations, hard to go below UI level.
Guerilla tests?
@arasb #uxsg
Why not…
136. Why not…
@arasb #uxsg
Remote testing? Too opaque to demonstrate good
research practices, may be hit/miss
based on the tasks and domain.
Can be too isolated and artificial.
Not very easy to ramp up in a week.
Lab studies?
Many potential confounding factors, hard
to get good behavioral insights.
Intercept surveys?
Hard to setup correctly across segments,
takes time for conclusive results.
A/B testing?
137. Why not…
@arasb #uxsg
Heuristics? No direct exposure to the user. Quality
depends too much on the domain and
the quality of the administering expert.
No users involved. Useful but
artificial way of finding problems.
Task analysis?
138. @arasb #uxsg
User research is asking deliberate questions
about a problem you care about, working
with your users methodologically to find out
answers and needs, discussing findings with
your team to design possible solutions.
141. @arasb #uxsg
We already know what our users want.
Research will delay our project.
We are not scientists.
We are engineers, not researchers.
Research is fairly expensive.
142. @arasb #uxsg
We cannot find the right
number of users.
This is not the right time
for research.
We really need these
features, we can do the
research-y part later.
The procedures are too
complex for our project.
We prefer to innovate.
But there is nothing to test!
We can figure it out in the
pilot phase/beta.
But we have Analytics?
143. @arasb #uxsg
Quick Do it in less than 10 days
Affordable No big budget planning
Correct Methodologically sound
Together Not over-the-wall
144. @arasb #uxsg
Field ImmersionUser Tracks
Benchmarking
Stakeholder Interviews
Snowball Meetings
Narrative Prototypes
Shared Analysis
Find a problem and
the questions
Work with users to
discover answers
Analyze & discuss with
the entire team
145. Everyone can do user research
Ask deliberate questions about a problem you care about,
Work with your users to find out answers and needs,
Discuss with your team to design possible solutions.
@arasb
/arasbilgen
Keep an open mind, keep your cycles short too.
Accept anyone who you come across this
journey as a fellow team member.
146. Thank you for your visit!
See my LinkedIn profile and follow me
on Twitter for my new presentations.
@arasb
/arasbilgen