This document discusses incorporating user testing and research into Agile development processes. It makes three key points: 1) Agile projects can miss overall user impact because changes are incremental, so user testing is important. 2) Simply observing users and having them think aloud while trying to complete tasks provides valuable insights into usability issues and why users behave the way they do. 3) Involving users regularly, even just testing mockups or concepts, and reviewing the results as a team can lead to major improvements in design and the user experience.

1. What your users really think
Incorporating user testing and research into Agile
Phil Barrett • Flow Interactive • Agile Africa 2014

2. UX design, research and
strategy
London & Cape Town
!
Since 1998
!
Hello.

3. You check out some new digital product.

4. You
“Meh.”

5. You
“Meh.”
Dear sir/madam
!
I recently visited your new website. I must
confess that at first, I found it a bit difficult to
understand the value you were offering.
!
I wasn’t sure if it was because I was not part of
your target market, or if you were still working
out exactly what was the best thing to offer.
!
I persevered and after a while realised that you
were actually providing a potential useful service.
I think it could be extremely profitable if you
simply make the following changes:
!
1.Ensure that I am not required to register before
Not you

6. Who sends you feedback?
No opinion Tentative opinion Strong opinion
Not using
Trying out
Casual user
Evangelist/Beta group
User

7. Who sends you feedback?
Feedback
Silence
No opinion Tentative opinion Strong opinion
Not using
Trying out
Casual user
Evangelist/Beta group
User

8. Missing feedback breaks Agile’s awesomeness
Build
Learn
Measure

9. Agile projects deliver gradual, incremental change
so it’s easy to miss overall user impact.
Flickr: Lars ploughman

10. Web analytics, tracking
What
do people do with
the product?
!
Hard data.
Split testing
What
will people do if we
try something else?
!
Experimentation.
The learn stack
UX testing
sessions
Why
do people do that?
!
Causes, inspiration,
direction, prediction
outside of known
situations.
!
Innovation.

11. F2F user testing gives you a blast of reality

12. I don’t need no stinkin’ feedback
• I know what people want
• My software is clearly good. I mean: look!
• Customers don’t know what they want
Flickr:DanielDionne

13. I know
what people want

14. You’re much better with software
than most of your users
Website tasks:
the slowest 25%
of users take
2.4 times
as long
as the fastest
25% of users
Your usersYou
http://www.nngroup.com/articles/variability-in-user-performance/

15. Econ: Considers, calculates,
makes optimal decisions
Human: Emotions, shortcuts
and irrationality
Real people are unpredictable

16. My software is good:
look!

17. Dan Ariely did an experiment.
With origami frogs.
They were hard to make and most people did a bad job.
How much would people bid for their own frogs?
And the frogs of others? And expert -made frogs?
Flickr:ToddJordan/Tojosan

18. Flickr:Nanimo
We think the things we make are expert quality.
• Average bid for expert-made frog: 27¢
• Average bid for own frog: 23¢
• Average bid by someone else for that same frog: ¢5c
Even when they are not.

19. “
The 9x effect
Executives, overvalue their own innovations...
Companies
overweight the new
product’s benefits
by a factor of
9
John T. Gourville
Harvard Business School
3
Consumers
overweight the
incumbent
product’s benefits
by a factor of
3

20. Customers don’t
know what they want

21. Humans are bad at imagining the future

22. Understand the jobs to be done

23. “To design an easy-to-use interface, pay
attention to what users do, not what they say.
!
Self-reported claims are unreliable, as are user
speculations about future behaviour.
Jakob Nielsen
NNGroup
Use observation

24. “There is a direct correlation between the
number of hours each team member
is exposed directly to real users and
the improvements we see in the designs.
!
It's the closest thing we've found to a
silver bullet.
Jared Spool
UIE

25. Let’s make it
incredibly easy

26. Doing a usability test
1.
Get an interface.
List key tasks.

27. 2.
Ask someone
new to try doing
the tasks, and
think aloud.

28. 3.
Don’t interfere.
Just write things
down.
!
Wait for 4 seconds.

29. 4.
Q. “Is this right?”
A. “What do you think?”
!
!
Be weird but friendly.

30. 5.
Record everything so
you and your team
can review the issues.

31. Open questions and storytelling
Do you like this? What do you think?
Do you understand this? What is this for?
Does this annoy you? How does this make you feel?
Do you want this? When will you use this?
Do you usually do this? Tell me the story of the last
time you did something like
this…

32. Get users from…
• The next desk
• The canteen
• Your forums
• Market research recruiters

33. Baby steps: Hall testing

34. You don’t need working code

35. Do usability tests in every sprint
Just tell the recruiter to get you “5 users
every thursday.”
Evaluate
Implement
Design and analysis Design and analysis
Implement
Design and anal
Evaluate
Implement
Evaluate Evaluate
Design and analysis
Implement

36. Test a mix of stuff
Past FuturePresent
Interviews
about past
experiences
Testing working
software
Testing
mockups and
concepts

37. Oh my goodness, stakeholders love it!

38. “The next step involved putting users in a room and
watching them use Obox. It was one of the most eye
opening experiences of our professional careers.
Watching a layman use your product will blow your
mind. You cannot even begin to imagine how your
users interact with it.
Obox blogged about their
usability testing experience
David Perel
CEO of Obox

39. § Get a team mate who likes
talking to people.
§ Get a target user.
§ Ask the user to do the 3
things the software is for.
§ Record it.
Your MVUT
Flickr: Lali Masriera/visualpanic

40. Thanks!
Phil Barrett • phil@userexperience.co.za • @philbuktoo
Flickr: Lars ploughman