Prague ACM SIGCHI presentation from 6/26/2006 about Remote usability testing by Karsten Skuppin

1. Remote Testing
Prague
June 2006

2. Remote Testing
quot;Historyquot; (I)
§ Usability testing for Otto
• Only specific areas / processes in the shop
• Not the quot;whole customer experiencequot;
§ Development of quot;seasonal testingsquot;
• Recruit participants that are about to buy something on otto.de
• Observe them just doing that in the labs
§ Open issues
• Testing is still not quot;realquot; enough
- Lab situation
- Perform tasks with possibly low involvement
- Not all relevant tasks in the scope of the lab test
Remote Testing page 2

3. Remote Testing
quot;Historyquot; (II)
§ Idea: Remote-Testing
• At home (quot;naturalquot; environment)
• Real involvement
• quot;Naturalquot; tasks (just what people want to do)
§ Challenges
• Tool must work without installation on PCs of users
• Tool must work without having to change anything on the server of
the client
• We must be able to quot;observequot; and quot;measurequot; (not only quot;askquot;)
Remote Testing page 3

4. Remote Testing
How does Remote Testing work?
Participants Proxy Server Web Site
Natural use of the target Passes inquiries on to
web site
Recalls all recorded
actions of the
respondents
Give free and SirValUse
scaled feedback Usability
Expert
Survey
Server Retrieves the
participants'
comments and
evaluations
Remote Testing page 4

5. Remote Testing
Variants of Remote Testing
§ Entirely free and realistic utilisation of the site.
Reported are quot;critical incidentsquot; that …
Critical • … trigger positive or negative sensations.
Incidents
• … prove to be crucial in reaching the target, e.g.
successful use of the search function or failure
due to complicated texts.
§ The respondents are confronted with tasks that
they have to master either with or without a time
limit.
Task-oriented
§ The participants themselves may assess whether they
have completed a task or not; the system is able to
recognise multiple target conditions.
Remote Testing page 5

6. Remote Testing
What does the respondent see? Critical Incident
Call up feedback Go back to
sessions questionnaire
positive feedback negative feedback end visit
Remote Testing page 6

7. Remote Testing
What does the respondent see? Critical Incident – positive feedback
You want to give us positive feedback?
Please describe the event or your observation!
What was it that you wanted to do just before the event you want to tell us about?
Please describe short and
precisely in your own words
what you wanted to do.
Example
Please describe the event or the observation and tell us what you liked about it.
Please describe the event or
your observation in a way
that a good friend of yours
would understand it.
Example
Does the event help you in doing what you wanted to do - as described in (1)?
very helpful somehow helpful not helpful at all
How much do you like this event?
very much somehow not at all
cancel submit … and continue surfing.
Remote Testing page 7

8. Remote Testing
What does the respondent see? Critical Incident – negative feedback
You want to give us positive feedback?
Please describe the event or your observation!
What was it that you wanted to do just before the event you want to tell us about?
Please describe short and
precisely in your own words
what you wanted to do.
Example
What happened? Please describe the event or the observation as detailed as possible.
Please describe the event or
your observation in a way
that a good friend of yours
would understand it.
Example
What were your expectations? Do you have any ideas what we could improve?
What did you expect from
T-Online? What or how
could we do better?
Example
Does the event prevent you in doing what you wanted to do - as described in (1)?
very much somehow not at all
How much do you dislike this event?
very much somehow not at all
cancel submit … and continue surfing.
Remote Testing page 8

9. Remote Testing
What does the respondent see? Task-oriented
Go back to
Task questionnaire
You're looking for a solution from T-Online in order to protect your PC from viruses, worms or other dangers that may be task done
connected with the Internet. Please look for some information regarding the quot;SecurityPackagequot;.
cancel task
Remote Testing page 9

10. Remote Testing
Was does the test head see? Screen shot, mouse traces, click points
Remote Testing page 10

11. Remote Testing
Example 1: Critical Incidents on otto.de
§ How is otto.de quot;reallyquot; being used? How do the
respondents proceed while browsing through,
searching or purchasing?
Questions § Which elements or rather functions stimulate or
obstruct purchase processes?
§ Which optimisation measures can be derived from
these insights?
§ 60 respondents were recruited offline.
• They were requested to use otto.de quot;in a normal
wayquot; for 6 weeks, but only via the Remote
system.
Method and
• While surfing they could report quot;critical
Implementa- incidents.quot;
tion
§ The remote system recorded all of the participants'
actions as well as the pages they visited.
• As a result valuable clues regarding optimisation
measures could be generated.
Remote Testing page 11

12. Remote Testing
Example 2: Task-oriented benchmarking
§ As a means of further developing the search engine,
one of our clients asked us to conduct a bench-
marking that covered 7 search interfaces:
• Where are the strengths and weaknesses of the
Questions individual searches?
• Which search options are employed for which
search?
• How long does it take to find a result and how
satisfied are the participants with it?
§ Altogether 175 User were recruited via a panel.
§ 75 of them employed 3 search interfaces each
Method and (rotated presentation) through the remote system.
Implementa- • Certain search tasks had to be completed (e.g.
tion quot;Please search for a picture of the Cologne
Cathedralquot;), while the remote system recorded
the times, the options employed, etc.
Remote Testing page 12

13. Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
Question: Did you find anything that would meet your requirements?
Search G 94%
Search C 88%
Search F 87%
Search
Success Search A 87%
All Tasks Search B 83%
Search D 82%
Search E 81%
Basis: persons who tested the respective search engine (N=75)
Remote Testing page 13

14. Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
Average search duration(in seconds)
Search F (N=65)
Frequent Users
Search D (N=62) Customers
Total
Search B (N=62)
Search
Duration Search C (N=66)
All Tasks Search A (N=65)
Search G (N=71)
Search E (N=61)
0
100 20 40 50 60 80 100
0
Basis: persons who successfully searched with the respective search engines
Remote Testing page 14

15. Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
When was the image search used in searching for a picture?
Search D
Search E Successful Search
Customers
Search C Total
Use of Image Search A
Search
Search G
Search F
Search B
0%
0 20%
20 40%
40 60%
60 80%
80
Basis: persons who tested the respective search engines (N=75)
Remote Testing page 15

16. Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
How often were search sections employed?
4 - 6 Times
6%
3 Times
15%
Never
Use of 44%
Search
Sections
2 Times
21%
Once
14%
Basis: total, per person during all searches (N=525)
Remote Testing page 16

17. Remote Testing
Example 3: Click Tracking
Remote Testing page 17

18. Remote Testing
Example 4: Testing of Variants
§ Four variants of a navigation scheme within an
online shop were to be tested:
• Which navigation scheme leads to satisfactory
results (users find products they would like to
Questions
buy)?
• Which navigation scheme can be used effectively
and efficiently (less clicks, less time, less
problems)?
§ Online recruitment of 1,000 users per variant.
• Users were asked to solve a task (same task for
Method and
each variant)
Implementa-
• Remote tool to measure effectiveness and
tion
efficiency
• Short online questionnaire to assess satisfaction
Remote Testing page 18

19. Remote Testing
Example 5: Control Quality of Portal
§ One of the biggest internet service providers in
Germany wants to control the quality of their
portals continuously:
• How do users assess content, usability, design
Questions etc. while using the portal?
• Is it possible for users to use the portal efficiently
and effectively?
• How can the quot;total user experiencequot; be improved
so that users intensify their usage?
§ Online recruitment of 2,500 users per month.
• 2,000 users solve 40 tasks to generate valid
measurements.
Method and
• 500 users surf the portal via the quot;critical
Implementa-
incidentquot; technique to generate ideas for
tion
optimization.
• Short reports every months, big reports every six
months.
Remote Testing page 19

20. Remote Testing
Research Questions
§ Critical incidents:
• Capturing actual user behaviour
• Adding quot;reasonsquot; and quot;qualitative feedbackquot; to your log files
§ Task oriented:
• quot;Hard figuresquot; needed
• Variant testing
• Benchmarking with competitors
§ General:
• Generate log files
• Quality control
• Target groups (e.g. B2B) difficult to recruit
• Target groups geographically wide spread
Remote Testing page 20

21. Remote Testing
Why Remote Testing?
§ The test setting provides the utmost validity.
§ Many respondents can participate within a short time.
§ Participants stemming from very diverse geographical
Advantages regions can be recruited.
over Lab Tests
§ Studies that require several appointments per
respondent can be realised without much effort.
§ Participants are in their usual surroundings;
they are not bound by fixed test dates.
§ quot;This tool made it possible for the first time to
conduct a test directly in the living room of
Otto.de users, i.e. in the environment in which
Customer
buying experiences are created and buying
Feedback
decisions are made.quot;
(Steffen Kehr, Customer Experience Management,
www.otto.de)
Remote Testing page 21

22. Remote Testing
Does Remote Testing have any disadvantages over lab tests?
§ Motivations and problems cannot be explored in
depth.
§ No opportunity to boost specific feedback.
Disadvantages
over Lab Tests § Sessions cannot be adapted to the wishes and
capacity of the participants during an interview.
§ No video tapes of the respondents' facial expressions
or gestures.
§ Remote Testing is complementary to qualitative
sessions.
Remote
Testing and § Problems identified by means of Remote Testing can
Lab Tests: be thoroughly investigated during a lab test.
a Useful
Combination § Problems identified in lab tests or by expert reviews
assist in further interpreting the results of task-based
Remote Tests.
Remote Testing page 22

23. Remote Testing
Automated Remote-Testing: User Experience Scorecard
Remote Testing page 23

24. Remote Testing
Automated Remote-Testing: User Experience Scorecard
Remote Testing page 24

25. Thank you for your attention!
SirValUse Consulting GmbH
Schlossstraße 8g
22041 Hamburg
Phone: +49 40 68 28 27 - 20
Fax: +49 40 68 28 27 - 20
www.sirvaluse.de
info@sirvaluse.de
Remote Testing page 25