Steve Krug presented on the topic of using participant picture-in-picture video (PPIP) during usability tests. While PPIP was long considered a good practice, Krug has come to believe it may be an unnecessary distraction. He conducted a survey that found the top reasons for using PPIP were to emphasize user misery and drive empathy, while the top concerns were privacy issues and potential distraction. Krug is interested in further research comparing test observer findings with and without PPIP to determine if it truly provides value or is a distraction. He recommends doing what works best for each individual context.

1. Steve Krug
UxPA Boston 2017
Questioning
Picture-in-Picture:
Why Showing the
Participant May Not Be
Such a Great Idea After All

2. The backstory
 Participant Picture-in-Picture (PPIP) means displaying the
usability test participant’s face in a small picture-in-picture
window.
 It’s long been thought of as a good thing
 Increases empathy with the user
 Helps convince the team (esp. management) that the product makes
users miserable and needs fixing
 I call it “Squirmcam” because historically it’s been about
highlighting user discomfort, pain, and failure
 Once difficult and expensive to do, it’s now easy
 Has become more or less standard practice

3. Backstory (cont’d.)
 I’ve come to feel that it may not necessarily be a good thing
 I think it's often an unnecessary distraction
 I’ve kept this opinion to myself except when asked
 Always prefaced it by saying “I know I may be wrong about this,
because I seem to be the only person who feels this way”
 When I decided to do this presentation, I was surprised to find
that some people whose opinions I respect felt the same way I
did
 As one of them said:

4. “My main issue about showing the participant is that
for me it really takes away from being able to
concentrate on the interface and what the user is
DOING and SAYING in relation to it.
It’s hard to keep concentrating anyway and easy to
get distracted, so why furnish a distraction that’s
NOT the main concern of the test?”

5. We’ve only got 45 40 minutes
 What this is about (done)
 A very brief history of PPIP
 My original list of pros and cons
 Survey results
 The questions that interest me
 My recommendations
 Q&A

6. A very brief history of PPIP video
 For a long time, PPIP was very difficult and expensive
 Had to be done in a “lab” with dedicated video cameras, tape
decks, video mixer, etc.

9. One-Way Glass!

10. A very brief history (cont’d.)
 Adding PPIP required a video operator and editor
 Very time-consuming
 Show of hands: Anyone ever edited analog video?

11. A very brief history (cont’d.)
 Around the turn of the century, things got dramatically better,
cheaper, easier
 Cameras > Webcams
 Scan converter > Morae, Camtasia
 Tape decks > Digital editing
 Boxes of videotapes > 1Tb pocket drive
 Video cables > Wi-fi, screen sharing software
 Thousands of $$ of hardware > Hundreds of $$ of software
 Do it if ROI warrants it > Capture it even if you may never use it

12. Typical ways PPIP has been used
 To create highlight reels to demonstrate participant “suffering”
 To create clips to use in presentations
 Part of a video record
 For viewing by people who couldn’t attend live sessions
 For possible future reference
 In the live view shown to observers

13. Two demo clips
 According to my proposal, at this point:
 “I’ll be showing two brief video clips with and without PPIP so session
attendees can experience the difference”
 Ultimately deemed infeasible for many reasons, like limited time
and your distance from the screen

14. Existing research
 Tried to do due diligence
 Apparently there isn’t much
 Here’s the most relevant study I could find
 Thanks to Chauncey Wilson, and others

15. BTW: Why don’t research
papers have dates?
(It’s a rhetorical question.)
http://download.techsmith.com/morae/docs/casestudies/pip-usafa.pdf

16. They took the obvious approach
 Showed two groups the same test videos
 One group with PPIP, the other without
 Asked them to identify usability problems
 Studied what problems each group reported and how seriously
they rated them
 Did a pilot test with experts, then final test with 24 human
factors students

17. Unfortunately…
 Main finding was that there was little agreement on problems
between and within groups
 e.g., “These results clearly show that expert agreement was less
than what would be expected by chance.”

18. Strong sense of déjà vu
 Studies done this way almost always seem to show/”prove” that
observers (even experts) don’t agree very much
 Rolf Molich’s Comparative Usability Evaluation (CUE) studies are
an amazing source of insights into what works

19. http://www.dialogdesign.dk/CUE.html

20. Strong sense of deja vue
 But they’ve “proven” over and over that if you show the same
usability test to experts, there won’t be much overlap in the
problems they report
 The “evaluator effect”
 I think it’s a flawed approach
 For reasons too complicated to fit in the margin here (Thanks, Fermat!)
 My conclusion:
 This one research study doesn’t tell us much

21. My initial pros and cons
 My original list, which I thought was pretty exhaustive
 …until I made the classic mistake of actually listening to users

22. Pros of using PPIP
 Adds sizzle
 The same way eye tracking can
 May get team members more excited about the process
 May get more bodies in the observation room
 May keep observers more engaged
 Keep them watching the screen, instead of their phones
 Can be persuasive
 It can help drive home the misery the UI/UX is creating
 Good way to get attention/support of higher-ups
 Improves chances of actually implementing changes

23. Cons
 Can distract observers from think aloud and screen activity
 Doesn’t reveal much that wouldn’t be otherwise obvious
 Complicates privacy issues
 Faces are recognizable
 How do you maintain control over clips?
 Will clients take same care you do?
 Best privacy solution is not to identify users at all; this is the opposite
 Adds more complexity to setup
 One more thing to break/keep track of

24. Cons (cont’d.)
 Can add to the participant’s stress
 I haven’t observed this myself
 (But I don’t do PPIP)
 My experience: they make a joke about recording at the beginning, then
forget about it
 Picture-in-picture obscures part of the screen

25. The survey
 Four questions plus a Comments field
 Ran for six days
 200 responses
 Mean time to complete: 3 minutes
 Deceptively advertised as 90 seconds
 But the questions were easy to answer
 Response data is available at
www.sensible.com/downloads/PPIP-survey-results.pdf
 Show of hands: Took the survey?

26. Caroline Jarrett!
 My forms and surveys hero
 She’s writing a book (Surveys that Work) for Rosenfeld Media
 “…just what you need to know about surveys: the least you need to
know to do a survey well, with the smallest amount of effort.”
 Helped pilot test and revise
 Analysed results

31. Caroline says:
Given our quite small sample size, we’ve
got an almost even number of people
across all the categories of PPIP use

35. What do you think are the three most important
reasons *for* using PPIP? (choose up to three)
0 20 40 60 80 100 120 140 160
Nothing in favor
Other
It makes observers more likely to attend live tests
Adds a dramatic “wow” factor
Good to use in clips to convince people who can’t/won’t
take time to observe whole tests
It keeps observers more engaged
Emphasizes user misery in a way that is more convincing
and likely to produce changes
It makes participants’ problems more visible and vivid
Encourages identification with and empathy for the user
Always / most of the time Sometimes Rarely / never

36. What do you think are the three most important
reasons *for* using PPIP? (choose up to three)
0 20 40 60 80 100 120 140 160
Nothing in favor
Other
It makes observers more likely to attend live tests
Adds a dramatic “wow” factor
Good to use in clips to convince people who can’t/won’t
take time to observe whole tests
It keeps observers more engaged
Emphasizes user misery in a way that is more convincing
and likely to produce changes
It makes participants’ problems more visible and vivid
Encourages identification with and empathy for the user
Always / most of the time Sometimes Rarely / never
Top three are all about
misery, empathy

37. What do you think are the three most important
reasons *for* using PPIP? (choose up to three)
0 20 40 60 80 100 120 140 160
Nothing in favor
Other
It makes observers more likely to attend live tests
Adds a dramatic “wow” factor
Good to use in clips to convince people who can’t/won’t
take time to observe whole tests
It keeps observers more engaged
Emphasizes user misery in a way that is more convincing
and likely to produce changes
It makes participants’ problems more visible and vivid
Encourages identification with and empathy for the user
Always / most of the time Sometimes Rarely / never
My “sizzle” notion
was a bust

38. What do you think are the three most important
reasons *for* using PPIP? (choose up to three)
0 20 40 60 80 100 120 140 160
Nothing in favor
Other
It makes observers more likely to attend live tests
Adds a dramatic “wow” factor
Good to use in clips to convince people who can’t/won’t
take time to observe whole tests
It keeps observers more engaged
Emphasizes user misery in a way that is more convincing
and likely to produce changes
It makes participants’ problems more visible and vivid
Encourages identification with and empathy for the user
Always / most of the time Sometimes Rarely / never
Caroline adds:
There’s quite a lot of agreement about the
good things and bad things about PPIP, no
matter how much they use it

39. What do you think are the three most important
reasons *for* using PPIP? (choose up to three)
0 20 40 60 80 100 120 140 160
Nothing in favor
Other
It makes observers more likely to attend live tests
Adds a dramatic “wow” factor
Good to use in clips to convince people who can’t/won’t
take time to observe whole tests
It keeps observers more engaged
Emphasizes user misery in a way that is more convincing
and likely to produce changes
It makes participants’ problems more visible and vivid
Encourages identification with and empathy for the user
Always / most of the time Sometimes Rarely / never
Caroline:
Respondents who don’t use PPIP saw slightly
more bad things (but not that much more)
Respondents who do use PPIP saw slightly
more good things (but not that much more)

41. What do you think are the three most important
reasons for *not* using PPIP? (choose up to three)
0 20 40 60 80 100 120
Did not choose any Against options
Other (please specify)
It makes life more complicated
It doesn’t produce better results
For some observers, it can be a distraction from…
Some people may refuse to take part if they’re…
Hard to do when testing mobile devices
Some observers may be biased based on the…
Showing people’s faces raises additional privacy…
It may make some participants uncomfortable or…
All / most of the time Sometimes Rarely / never

42. What do you think are the three most important
reasons for *not* using PPIP? (choose up to three)
0 20 40 60 80 100 120
Did not choose any Against options
Other (please specify)
It makes life more complicated
It doesn’t produce better results
For some observers, it can be a distraction from…
Some people may refuse to take part if they’re…
Hard to do when testing mobile devices
Some observers may be biased based on the…
Showing people’s faces raises additional privacy…
It may make some participants uncomfortable or…
All / most of the time Sometimes Rarely / never
I was glad to see so much
concern about privacy

43. What do you think are the three most important
reasons for *not* using PPIP? (choose up to three)
0 20 40 60 80 100 120
Did not choose any Against options
Other (please specify)
It makes life more complicated
It doesn’t produce better results
For some observers, it can be a distraction from…
Some people may refuse to take part if they’re…
Hard to do when testing mobile devices
Some observers may be biased based on the…
Showing people’s faces raises additional privacy…
It may make some participants uncomfortable or…
All / most of the time Sometimes Rarely / never
Not much support for my
distraction theory

45. 50 people added comments
 “The %age of my recommendations that were adopted went up
by 60% when I started using videos”
 “It's also invaluable to teach stakeholders who their users really
are (breaking assumptions - for example, an early adopter can't
be an older woman... actually, yes they can! See for yourself!)
 “Creating video highlights is a time luxury that many (most?)
researchers don't have.”
 “I just do it because it's default behaviour in my recording
software.”
 “My teacher, umm you, said it wasn't that necessary and after
these years I agree.”

46. Questions I find interesting
 Is it possible that showing observers picture-in-picture video is
actually a distraction?
 There’s a lot to focus on when watching a usability test
 What’s happening on the screen (clicking, scrolling, text entry, etc.)
 What the user is saying about what they’re thinking, and how they’re
saying it
 Your own mental model of what the participant is doing and thinking

47. Audio trumps1 video
 You can usually infer emotional state from voice
 Clear audio reduces effort, people can focus longer
 My priorities, in order
 High quality audio
 Large enough screen image
 Participant face video (distant third)
1 Pardon the expression

48. “But won’t you miss non-verbal clues?”
 I think you get the same information from audio
 Facial expression and body language clues are almost always
accompanied by tone-of-voice clues and/or pregnant pauses
 A good facilitator will notice and call out important aspects of
nonverbal behavior
 “What are you thinking?”

49. We’ve got to consider privacy
 It worries me that some people don’t seem to take privacy all
that seriously
 Is it possible to distribute clips and still maintain privacy?
 Perv bars
 No names, please (should be rule in any case)

50. What are you in it for?
 I consider myself a bit of an outlier
 I’m in it for insights
 My charmed consultant life
 I can just peddle insights and good ideas
 I live with the Eloi and play in the sun

51. Members of the gentle surface-dwelling Eloi race - The Time Machine (1960)

52. Members of the ruthless subterranean Morlock race - The Time Machine (1960)

53. Eloi Yvette Mimieux - The Time Machine (1960)

54. Yvette Mimieux – “Tyger, Tyger” episode of Doctor
Kildare (1964)

55. What are you in it for?
 But many people are in it for persuasion
 You may live in a Morloch world, where powers of persuasion are
required
 PPIP may be a much-needed, powerful tool for you

56. Anyone looking for a research project?
 Comparative study of observer findings of the same test with
and without PPIP
 Eye tracking of some observers to determine time spent
watching PPIP
 I’d be happy to help
 But you’d have to tolerate my bias for qualitative research

57. In conclusion…
 Sometimes I wish I hadn't pulled on this thread
 Who knew recording participants could be so complicated?

58. What I recommend
 Do whatever works in your context
 The same way I wouldn’t tell you not to do eye tracking, if it furthers
the cause of usability in your organization
 But do it consciously, with awareness of the tradeoffs
 Be judicious about privacy issues
 Seeing the participant does give you an additional useful sense of who
they are
 Possible alternative: Show participant to observers briefly at beginning
of session
 Check out these two books:

61. One final show of hands, please
 Have you changed your mind at all about PPIP?

62. Photo © Jeff Jeffords www.divegallery.com

63. Thanks for all the fish
 Send any questions, feedback, gripes to
skrug@sensible.com
@skrug
or visit
www.sensible.com

64. Q&A

65. © 2017 Steve Krug