Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Easy and affordable user testing - Workshop


Everyone wants to make intuitive products. But you can’t really know they’re intuitive, if you haven’t tested them with actual users. So why don’t we user test more often? It might be because we think we’re lacking the time, budget, equipment or simply the knowledge to do it. But that doesn’t have to be true.

Tools and methods: Get to know 7 easy and affordable techniques to do user testing
Planning and prioritising: When and what should you test?
Tasks and moderating: How do you make sure you’re not leading users during the test?
Recruiting: How many people do you need to test with, and where can you find them?
Observation: How should you interpret what you see and take notes during a user test?
Analysis: How can make sure your usability findings lead to actual improvements?

Easy and affordable user testing - Workshop

  1. 1. Easy & affordable
 user testing Ida Aalen / @idaaa CPO, Confrere Workshop
  2. 2.
  3. 3. Task 1 Read the title and the intro. Write down 1-5 questions you expect will be answered on this page.
  4. 4. Task 2 Read the text. • Were your questions answered? • Is there content that could be cut or should be expanded upon?
  5. 5. #1 Micro-testing content. • Time: 30 min+ • Price: Mostly free
  6. 6. Question 1 Read the title and description. Write down 3-4 questions you expect will be answered on this page. Don’t peek!
  7. 7. Feedback on needs, not grammar and typos
  8. 8. Feedback on needs, not grammar and typos
  9. 9. • What equipment is supported? • Can this service be used on mobile devices? • How can I know that it is safe? • What is the flow like, step-by- step, when setting up a call? • How much does it cost? • How do you embed/link this on your own site?
  10. 10. User testing can be
 easy & affordable.
  11. 11. Why do
 user testing?
  12. 12. User testing is
 real users solving real tasks.
  13. 13. Do you recognise this icon? 
 Survey with 4273 respondents, 2014 Mobile: 53%
 said no Desktop: 73%
 said no
  14. 14. …but I can’t see the football…
  15. 15. Colleagues
 are not your users.
  16. 16. Designers
 are not your users.
  17. 17. Developers
 are not your users.
  18. 18. 1 in 3
 problems found
 by experts are
 false alarms 1 in 2
 user problems
 are overlooked
 by experts
  19. 19. Only users can let you know whether it’s intuitive. But how many users?
  20. 20. 5
  21. 21. N (1-(1-L)n)
  22. 22. 0% 25% 50% 75% 100% Number of test subjects 0 3 6 9 12 15 Shareobservedproblems Small and frequent tests are better than bigger and infrequent tests
  23. 23. User testing • …it’s the only way to know if the design truly is “intuitive” • …you’re taking big risks when you’re not testing • …it improves communication with stakeholders
 And it’s cheaper than you think :)
  24. 24. #2 Information architecture • Time: 30 min+ • Price: Free / $99-199 per month
  25. 25. #3 First click. • Time: 30 min+ • Price: Free / $99-199 per month
  26. 26. Tasks.
  27. 27. Finding possible issues Customer service and receptionists What are people calling and asking about? Website analytics Where do users drop off? Where do they spend more time than expected? Search logs Are there things that people aren't able to find? Which words do they use?
  28. 28. You should test things that… • are done frequently or by many users • has serious consequences if users fail with their task • are hangups within the organization, but need to be determined if they're actual issues
  29. 29. What we want to find out: Is it clear to physicians that they can use Confrere for video appointments?
  30. 30. Our question ≠
 User task
  31. 31. Open & explorative What would you do to find a video calling service that’s suitable for physicians? Specific & measurable Could you find the page which explains how Confrere is HIPAA compliant? Partially open Go to What would you do to figure out whether this is suitable for physicians?
  32. 32. Open & explorative • If you want to find more problems • It’s ok to give different tasks to different people • More realistic, but also more time- consuming Specific & measurable • When you know there’s a problem, but you want to figure out why • Easier to measure and compare • Can be leading Partially open
  33. 33. Phrasing the tasks • Don’t use the exact same phrases as on the website • Give the users a problem to solve, rather than asking them to “find” something • Adapt the tasks to each person, to make them more realistic
  34. 34. What do we want to know? “Is it clear to physicians that they can use Confrere for video appointments?”
 How do we phrase it? “Go to What would you do to figure out whether this is suitable for physicians?”
  35. 35. Intro • User’s rights and informed consent • Introductory questions / warm-up
 Tasks • The user tries to solve this, while the moderator sits next to them
 Closing • General feedback • Thanking the user Typical duration
 15-60 min
  36. 36. Time schedule • You’ll always underestimate how long the tasks take • Begin with 2-5 tasks, and do a trial test • Prioritize the tasks, in case you don't have time to get to all of them • Make sure you have time to test the equipment • Breaks between each test: At least 15 minutes of break time per hour!
  37. 37. Moderating.
  38. 38. Help the person to relax Make it easier for observers to understand Make sure we follow the test plan
  39. 39. Help the person to relax.
  40. 40. “Would you like something to drink?”
  41. 41. Checklist before beginning • Welcome the person, help them feel comfortable • Go through rights and consent form, make it clear that we’re testing the system, not you • Explain the “think out loud”-method • Ask if the person has any questions before you start • Remember to turn on the recording!
  42. 42. Help the person think out loud
  43. 43. What are you thinking about?
  44. 44. I’m looking for the menu… Help the user think out loud and ask open questions
  45. 45. What happens if I click here? What do you think might happen if you click there? Make sure you’re not helping the user
  46. 46. How to help someone think out loud • What are you looking for now? • What are you thinking about now? • What did you expect would happen just now? • …but don’t interrupt
  47. 47. How do you lead a test, without leading the user?
  48. 48. I’ve made the design for the new website and I’d love to hear what you think! The team that designed the site needs help figuring out if it's really working.
  49. 49. You’re really good at this! You’re solving all the tasks! Thank you! Let’s move on
 to the next task.
  50. 50. It would have gone so much faster if you had just clicked the menu-button. There's really big room for improvement here. Thank you for helping us identify this!
  51. 51. Sigh…
  52. 52. The user
 is there to help you
  53. 53. #3 Get out. • Time: 1-4 hours • Price: Snacks
  54. 54. It’s this easy:
  55. 55. (Translated by Google)
  56. 56. What did you find?
  57. 57. Analysis in 1-2-3 Is there anything you observed… …across several users on the same task?? …across several tasks? You start doing the analysis between each individual test.
  58. 58. Observation
 and notes.
  59. 59. Don’t listen. Observe.
  60. 60. Rationalization 
 Michael S. Gazzaniga: “The Interpreter”
  61. 61. Differentiate between… Observation What the person did and said during the task Interpretation Why you believe the person did what they did Improvements What you believe could be a solution to the problem
  62. 62. Example Observation • The user did not click the “Buy”-button Interpretations • The button should be placed elsewhere • The button has the wrong color or shape • The microcopy “buy” sounds too committing • The user didn’t think they really were supposed to “buy” since it’s just a test
  63. 63. Quantitative Qualitative Self-reported Observed Surveys A/B-testing User testing Interviews Focus groups Field studies Google Analytics Key word analysis “Was this helpful?” Crawling
 social media
  64. 64. Read this!
  65. 65. How to take notes? • Remember the purpose of the task - don’t write a novel on what each person does • Note which words the user is using • Remember, it's mostly about what they're actually doing and saying while they're doing it
  66. 66. Paper sketch Paper prototype Wireframes Sketch / Photoshop InvisionApp HTML prototype without styling HTML&CSS&JS prototype Interactive Flat Rough Detailed Functional Implemented Text
  67. 67. #5 Scrolling, clicking and hovering. • Time: Done in 30 min • Price: Free (e.g. Hotjar + Google Analytics)
  68. 68. #5 Was this helpful? • Time: 15 min+ • Price: Often free
  69. 69. #4 Lab test on a budget. • Time: 1 day for planning and recruiting, 1 day for the actual test • Price: Rewarding users
  70. 70. (Translated by Google)
  71. 71. “Hugging”
  72. 72. If you need observers: Test machine Observation + Tool for
 screen sharing+
  73. 73. Informed consent • They can opt-out any time • What data are you gathering and why • For what and when will you use it • When will you delete it • Check local rules/laws KONTAKT OSS +47 22 42 46 42 BESØK OSS Stenersgaten 8, Oslo Dine rettigheter som testperson 1. Det er ikke du som testes, men løsningen. Det er derfor ingen krav til prestasjoner eller ferdigheter. 2. Det er frivillig å delta og du kan avbryte testen når som helst 3. Det blir sendt bilde av skjermen og testsituasjonen til et annet rom 4. Din anonymitet sikres ved at det ikke brukes navn eller annen informasjon som kan koble personen til notatene 5. Opptak av skjermbildet vil kunne bli vist i møter eller lignende etter testen er ferdig, men det vil ikke være fokus på hvem du er. Videoene vil kun være til internt bruk i prosjektet og vil slettes etter tre måneder 6. Du skal behandles høflig og med respekt Jeg bekrefter å ha fått den informasjon som er angitt ovenfor og at jeg gir tillatelse til at opptak av skjermbildet kan bli vist til andre. _________________________ _________________________ UNDERSKRIFT DATO KUNDE: PROSJEKT:
  74. 74. 20-30 min break between each test
  75. 75. When do you need observers? • If you need to align stakeholders • In general, the more observers, the better, because different people notice different things • Bring your teammates from different disciplines (e.g. developers) • Observers should stick to observing, and not introduce themselves to the users
  76. 76. Analysis and prioritisation.
  77. 77. The analysis must be done as soon as possible, by the people who did the user test.
  78. 78. Which of our findings or observations should be turned into to-dos? What’s relevant for our further work?
  79. 79. Documenting and sharing your findings
  80. 80. As a tool for your team
  81. 81. Put screenshots in InvisionApp in advance Add observations between (or during) each test
  82. 82. Connect InvisionApp to Slack, and your teammates can follow along
  83. 83. Vise alvoretWake-up calls
  84. 84. Innhold
  85. 85. A highlights video usually does the trick.
  86. 86. Prioritizing number of users consequences other sources cost
  87. 87. Make sure something happens with your findings • Each (prioritised) usability issue needs a to-do • Each to-do must be assigned to a specific person
  88. 88. Do it together!
  89. 89. Summarizing and prioritizing in Google Sheets or AirTable
  90. 90. Recruiting.
  91. 91. Selection criteria • Diversity • Language skills • ICT skills • Target group criteria • Background knowledge or experience with your service
 Exclusion criteria • People with too much knowledge about your service • People working with development, design, web etc Stricter criteria
 Harder to recruit
  92. 92. Recruiting tips • Friends of friends, social media, newsletters • Go to where they are • Run it outside of regular work hours • Be creative about rewards • Lower your standards
  93. 93. Avoiding drop-outs • Send e-mails or texts with very clear instructions for how to get there, sign them personally and add your phone number • Send a text the day before or the same day to remind them • Recruit one more person than you think you actually need
  94. 94. 7 methods: 1. Micro testing content 2. Information architecture 3. First click 4. Get out 5. Lab test on a budget 6. Scrolling, clicking and hovering 7. “Was this helpful?” usertest
  95. 95. 7 methods: 1.Micro testing content 2.Information architecture 3. First click 4. Get out 5.Lab test on a budget 6. Scrolling, clicking and hovering 7. “Was this helpful?” usertest
  96. 96. 7 methods: 1. Micro testing content 2. Information architecture 3. First click 4.Get out 5. Lab test on a budget 6.Scrolling, clicking and hovering 7.“Was this helpful?” usertest
  97. 97. 7 methods: 1. Micro testing content 2. Information architecture 3. First click 4.Get out 5.Lab test on a budget 6. Scrolling, clicking and hovering 7. “Was this helpful?” usertest
  98. 98. You learn so much more by meeting people face-to-face!
  99. 99. Thank you! Ida Aalen Chief Product Officer