3. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Evaluating a product or
service by testing it with
representative users.
3
Usability
Testing
4. Participants will
try to complete
tasks while
observers watch,
listen, and take
notes.
Goals:
• identify any usability
problems
• collect qualitative and
quantitative data
• and determine the
participant’s satisfaction
with the product
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
4
5. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
5
6. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
6
7. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
7
9. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
9
10. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Confirmation Bias
Tendency of people to seek or
believe information that
confirms their existing beliefs.
10
11. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Acquiescence Bias
Participants agree to whatever the interviewer
presents to them in order to:
• be liked
• pass the “test”
• earn their stipend.
“This was
really easy.”
“I understood
that clearly.”
“10 our of 10!”
“When can I
buy this?!”
11
12. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Central Tendency Bias
People are hesitant to
commit to extreme positions.
12
(Find me later if you want
to talk about all of the other
problems with this survey.)
13. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Social Desirability Bias
People lie when you ask
them about sensitive topics:
• sexual habits
• drug, alcohol, or tobacco
use
13
… and not-so-sensitive
topics:
• internet/phone use
• exercise & eating
habits
• compliance with rules
14. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Hawthorne Effect
When people know they are
being observed, their
behavior changes.
Additional observers
magnify this effect.
14
15. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Evaluator Effect
Different evaluators find
different issues.
15
16. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Selection Bias
Those who
participate in a
research activity are
relevantly different
from those who
choose not to or are
unable to
participate.
16
22. 22https://www.userfocus.co.uk/articles/usability_test_plan_dashboard.html
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Determining the
Research Questions
How well or poorly do
people understand
Feature X?
Which elements of the
checkout process are
confusing?
How frustrated do people
feel after interacting with
this product?
Do people understand the
different pricing options?
What questions do users
still have after reading
about the app?
27. 27https://www.userfocus.co.uk/articles/usability_test_plan_dashboard.html
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Testing
Where
- research facility
- conference room
- remote (URL)
- remote observation?
When
- date(s) and times
- specify time zone
Reporting
Where
- conference room
- remote (URL)
- storage of report
When
- date(s) and time(s)
32. 32
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
#NotALawyer
Know the law
Geographically and
situationally dependent.
Special
considerations
for children and
some other
populations.
33. 33
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Informed Consent
Participants get all
information in a form
they can understand.
• Purpose of research
• Right to decline/withdraw
• Risks/discomfort to them
• Benefits to participant
• Limits of confidentiality
• Incentives
• Contact information for
follow-up questions
34. 34
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Confidentiality and
privacy
• Preserve anonymity
• Never use surnames
(in session or report)
• Blur faces if necessary
• Honor agreements re: video
and data distribution
• Take practical security
measures with recruiting
and payment information
and stored recordings
35. 35
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Participant comfort
Physical and
emotional
40. 40
Confirmation Bias
Tendency of people to seek or
believe information that
confirms their existing beliefs.
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Recruiting
• choosing highly
experienced participants
Tasks
• easy tasks
• no stress cases
• leading phrasing
Observation
• selective listening
• downplaying or dismissing
issues
41. 41
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Acquiescence Bias
Participants agree to whatever the interviewer
presents to them in order to:
• be liked
• pass the “test”
• earn their stipend
Self-blame
High rating scores
You can pay
them first!
42. 42
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Central Tendency Bias
People are hesitant to
commit to extreme positions.
1 2 3 4 5
1 2 3 4 5 6 7
rating scales
comments
43. 43
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Social Desirability Bias
People lie when you ask
them about sensitive topics:
• sexual habits
• drug, alcohol, or tobacco
use
44. 44
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Hawthorne Effect
When people know they are
being observed, their
behavior changes.
• Hide observers in a separate room
• Downplay their numbers
• but still must disclose
45. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Evaluator Effect
Different evaluators find
different issues.
45
Use multiple evaluators.
46. Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Selection Bias
Those who
participate in a
research activity are
relevantly different
from those who
choose not to or are
unable to
participate.
46
47. 47
Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
danielle@dgcooley.com dgcooley.com