Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

UXPin Agile UX
Virtual Summit
June 15, 2017
Carol Smith
@carologic
Faster Usability Testing
in an Agile World

IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Design at IBM Watson
Design Thinking & Agile at IBM with Vera Rhoads
16th June 2017, 2:00-3:00 PM PDT

• Researcher and interaction designer on each squad
• Visual designer often shared
• Designers Consult on other teams as needed
Watson - Squads include UX Designers

• Iterative
• Repeatable
• Incremental
• Get to 80% confidence
Light Design - Lean UX

• Just enough
• Identify and solve problems
• Use each study to pick up information
• Think ahead (not make ahead)
– Concept Cars (vision)
– Know the North Star
No BDUF* at Watson
*Big Design Up Front

• Fit testing into Agile process
• Never too early - whatever is ready
• Regular testing
Image of sign, created by IBM’er Thomas J. Watson
http://www.newyorker.com/cartoons/bob-mankoff/computer-games

• Representative users
• Doing real tasks
• Being observed
• Using prototypes
or live products
Usability Testing is…
http://creativecommons.org/licenses/by-sa/2.0/
http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/
http://www.flickr.com/photos/raphaelquinet/

• There is a difference
• User testing infers testing people
– Can the participant figure out our design work?
• Usability testing puts emphasis on what is being tested
– Reduces the focus on the participant’s performance
– Is the product successful?
Usability NOT User Testing

• All interfaces have usability problems
• Limited resources to fix them
• More problems than resources
• Less serious problems distract
• Intense focus on fixing most serious problems first
True Statements
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug

Parallel-Track
Workflows
(How to Think Ahead)
a.k.a. Staggered Sprints

• Developer Track:
– Focus is on production
– Whole team (including UX)
• UX Track (Interaction Designer Track)
– Focus is on user contact
Agile Design Timing: Parallel Tracks
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
UX

Iteration 1
UX

Iteration 2

And so on…

1) Development work
– Maximize velocity to make working software
– Goal: Working software
2) Discovery work
– Learning velocity - fast learning and validation
– Goal: Learning
Reality Check – 2 Types of Work
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/

Dual Track Development is not Duel Track
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/

• Between the two tracks
• Essential for success
• Not just hand-offs
• All at once – two tracks, not two teams
• All work will not move from discovery to development
Constant Communication
From Tutorial: Making UX Agile and Lean at UXPA 2016
by John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean

• Pros
– Space is made for discovery work
• Cons
– Can lead to BDUF and Waterfall
– Not a solo sport - need a team
Pros and Cons - Parallel/Staggered work

Balancing Act
• Understand users, context, etc.
• Create personas, mental models, etc.
• Prepare for story mapping
and other sessions thoroughly
Strive for
UX Best
Practices
• Engineers need designs
to develop
• Research cannot continue forever
Meet
Production
Needs
Agile requires leaner methods
Time box and focus on outcomes
Slide from Tutorial: Making UX Agile and Lean at UXPA 2016
By John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean

• Continuous improvement
• Continue to prioritize Dev work
• Start small with Discovery work in next sprint
• May need a Design Spike
Existing products/teams

What to Test

Learn from whatever is ready…
• Concepts, rough ideas,
brainstorming
• Work in Progress
• Prototypes - make to learn
– Clickable (code is optional)
• Websites, mobile, products,
services
• Competing designs (A/B
testing)
• Comparative studies
• Conduct user research

• Clickable
– Easy and quick to create
– Easy to change
• Real Code
– Risky – more time invested
– Great if it’s the right solution
• Not paper - participants may misunderstand
Clickable or Code?

When should I test?
• Early! Any Stage of process
• Anywhere (conference room,
remotely)
• Anytime (un-moderated)
• Realistic test environment
Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/

Testing Methods

• What do we need to know?
• What gets us to 80% confidence/disproves?
• Bare minimum of effort?
• Place higher value on repeatable formative studies
Adapting Usability Testing Methods

• Consider budget, resources
• Time
– Recruiting
– Facilitating
– Analyzing
Scope Effort

Small Scope is Better
• 5-7 participants uncover
~80% of major usability problems
• Don’t try to be statistically
significant
• More participants
- diminishing returns
Jakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users.
March 19, 2000. http://www.useit.com/alertbox/20000319.html
Number of Test Participants
Participants are representative of specific user group and are doing representative tasks
More participants take more time, cost more money - significantly diminishing returns.

• Moderated
– In Person
– Remote (Web Meeting)
• Un-moderated (online tools)
Testing Options

• 3 + participants (tie breaker)
• Short sessions
– Moderated: 30 minutes to 1 hour each
– Unmoderated: Under 20 minutes
Make it Quick

• Pros
– Get to “why” by asking follow-up questions
– Work with them longer
– Save or end session as appropriate
• Cons
– Harder to recruit
– Takes longer to plan
Moderated

• Pros
– Great for quick answers - complements moderated testing
– Easy recruiting
• Cons
– Recruiting very relaxed
– Professional participants
• Participants camera and internet only
Unmoderated

Remote Testing Options
Moderated Un-Moderated
Researcher is present Researcher not needed during study
Same effort as in-person Minimal effort
Limited number of participants
per day (3-6)
High number of participants per day
Immediate feedback Test data compiled usually within
2 days
Get to the “Why?” by asking
questions
Reduced or no ability to ask
questions

• No travel
• Easy to set up
• Conducted from almost anywhere
• Option to see participants actual environment
• Drawback: Not as high quality as in-person
Benefits of Remote

• Robust computer
• Screen sharing software
• Fast Internet on both ends
– May require separate phone line if no VOIP
• Participants camera
– People are becoming more comfortable
– I still do not use
Remote – Moderated Requirements

More on Remote Testing
• Getting the most out of remote
research and testing
– http://interactions.acm.org/archive/view/march-april-
2017/getting-the-most-out-of-remote-research-and-testing
– ACM Interactions Magazine
by Carol Smith
Getting the most out of remote research and testing. By Carol J. Smith. 2017.
Interactions, 24, 2 (February 2017), 82-84. DOI: https://doi.org/10.1145/3038225
http://interactions.acm.org/archive/view/march-april-2017/getting-the-most-out-of-
remote-research-and-testing

• In-person (office or participant location) or remote?
• Prototype/organization limitations (Online? Clickable?)
• Number of participants, observers?
• Logging and video editing (time on task, highlights)?
• Surveys before or after?
• Eye tracking?
Software Tool Considerations

Software Tool Considerations
Meeting
Software
ModeratedUn-Moderated
Complex
Simple

• Nice
– Are your users local?
– Opt for large screen TVs/Cameras vs. two way glass
• Recommend laptop/portable lab
– Can be semi-permanent
– Ready when you are (minimal setup)
Formal Lab?

Make it Repeatable

“Teams should stretch
to get work into that day’s test
and use the cadence to drive
productivity.”
- Jeff Gothelf
http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-
with-agile/

Regular and Early Testing
• Reduces risk and chance
of major issues later
• Priority needs and problems
get priority
• Don’t sweat the small stuff

• Small focused tests
• Reduced recruitment time
• Once per Sprint
• Same day(s) mid-week (avoid Mondays and Fridays)
Bring it On!

• Team becomes:
– accustomed to steady stream of qualitative insight
– insight encourages quick decisions…line up with business
and user goals
Why Regular?
Adapted from Jeff Gothelf - http://blog.usabilla.com/5-
effective-ways-for-usability-testing-to-play-nice-with-agile/

• Problems we will find while usability testing
• Research to inform personas
• Design/UX debt
• Advocate to get prioritized (UX rep should be in planning)
Design Stories

Managing Testing

• Select what to test and/or additional work
• Create guide
• Determine recruit and automate where possible
Planning is Tight

• Screener
– Technology use/experience
– Knowledge of topic
• Guides/Scripts
• Consent Forms
• Data Collection
Create Reusable Templates

• Test & Observation Rooms
– Use repeat calendar feature
• Any location will do
– Conference rooms
– Offices
– Quiet corner of cafeteria
Pre-Book Rooms Quarterly

Recruiting

Hire a Recruiter
• Allows you to focus
• Good recruiters:
– find right participants.
– give regular updates.
– take care of directions,
confirmations, incentives, etc.

• Email, phone, etc.
• Create a panel (large recruits, less often)
– Customer lists, user groups
– Schedule as needed
– Use cautiously
• Internal resource who REALLY understands who you need
Do it Yourself

Use Site to Recruit
Ethnio: https://ethn.io/

• “We are all only temporarily able-bodied.
Accessibility is good for us all.”
-@mollydotcom at #stirtrek 2011
• Know the law, build for the spirit of accessibility
– U.S. Section 508 - https://www.section508.gov/content/learn
– WCAG 2.0 - https://www.w3.org/WAI/intro/wcag.php
Include people with disabilities

• Keep it easy
– In Person: Gift cards from retail stores
– Remote: Online Gift Cards
• More complex
– Product credits
– Swag (don’t bother)
Honorariums

• Make team aware
• Invite everyone
– Can watch remotely
– Recurring meeting invites for stakeholders
Usability Testing Day!

Observing Studies

• Stakeholders and decision makers
– PO, PM, Dev, etc.
• Greet participants (in person sessions)
• Take notes and share them
– Create a template if helpful
• Help operate software/recording equipment
Team Members Are Active Observers

• Pro’s
– Gain empathy for users
– Understand struggles and what works
– Quick decisions can be made
• Cons
– Time is limited
– Not all sessions are great (but most are!)
Observers

Notetaking
• Keep it simple
• Paper or Computer
– Ask participant if bothered
by typing
– Consider if you can pay attention
(turn off all distractions)
Issue P1 P2 P3
Search Used Yes No No
Widget 1 Used N/A Used –
unsure
about
Task 1 Notes 3 – easy 2 – needed
help
3 – easy
Task 2 Notes 2 – needed
help
2 – easy 2 – needed
help
help
3 – easy Ran out of
time
help
3 – easy Ran out of
time

Analysis, Debrief
and Useful & Usable
Recommendations

• Occurs without full analysis when stakeholders have watched
tests
• If not: Quick analysis to quick decisions
Debrief with Team

• All decision makers MUST be present
• Stay on Topic
• Be Constructive
• Don’t get distracted by small problems
• Intense focus on fixing most serious
problems first
Guidelines for Debrief

• Identify top 5 or 10 most serious issues
– Top 3 from each observer’s list
– Prioritize from lists
– Commit resources for next sprint
– Stop
Goal
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug

Excel - Find Patterns in Data
Learn more: I’m Sorry. I Can’t. Don’t Hate Me:
The Post-It Breakup by Kyle Soucy at IA Summit 2017
https://www.slideshare.net/usableinterface/im-sorry-i-
cant-dont-hate-me-the-postit-breakup

• Look for patterns
• Read “between the lines”
• Know what you’ve got
– Sort, reorganize, review, repeat
– What refutes your expectations?
– Surprises?
– Outliers?
Transform Data

Don’t be a tree killer
Image of stack of paper:
https://commons.wikimedia.org/wiki/File:Stack_of_Copy_Paper.jpg
X Marks the Spot https://openclipart.org/detail/193050/x-marks-the-spot

• Bulleted list
– Who will use information and how?
– Share via common tools (wiki, Box, etc.)
• Include
– Goal of study
– High level findings and recommendations
– Link to guide and analysis (tasks attempted, actual
performance, etc.)
Just enough

• Identify stories related to findings
• Add findings/evidence/recommendations in comments
on related stories in repo
• Tangential findings/evidence/recommendations - also note
in comments
Tie to Stories in Repo

• Small iterative changes
– Make it better now
– Don’t break something else
• Take something away
– Reduce distractions
– Don’t just add – question it
Tweak, Don’t Redesign
Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding
and Fixing Usability Problems, by Steve Krug

• Fit testing into Agile process
• Never too early –
whatever is ready
• Regular testing
https://www.flickr.com/photos/fireatwillrva/5823369990

Contact Carol Smith
in/CarolJSmith
@Carologic
slideshare.net/carologic

Resources, References, etc.

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

Similar to Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin (20)

More from Carol Smith

More from Carol Smith (20)

Recently uploaded

Recently uploaded (20)

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin