SlideShare a Scribd company logo
1 of 37
Webinar:
How to Conduct UX Benchmarking
May 2nd, 2012
Intros




          Alfonso de la Nuez             Kim Oslob
         Co-Founder & Co-CEO     Sr. Client Services Director



                    Twitter Hashtag #uzwebinar
Overview


 What we mean by UX Benchmarking and why it’s important?

 How to conduct UX Benchmarking studies, using remote
  unmoderated usability testing (aka task-based surveys)?

 What are some of the UX metrics you should look for?

 How to measure the quality of a website’s user experience across
  various competitors?

 A case study: American Airlines vs Delta Airlines Websites
What is Benchmarking?


Definition of Benchmarking:
 The process of comparing the business
  processes and performance metrics
  including cost, cycle time, productivity,
  or quality to another that is widely
  considered to be an industry standard
  benchmark or best practice.

 It provides a snapshot of the
  performance of your business and helps
  you understand where you are in
  relation to a particular standard.
What is UX?


Definition of UX:
 It’s about how a person feels about using a
  system. It highlights the experiential,
  affective, meaningful and valuable aspects
  of HCI and product ownership, but it also
  covers a person’s perceptions of the
  practical aspects such as utility, ease of
  use and efficiency of the system.

 User experience is subjective in nature, as
  it is about an individual’s feelings and
  thoughts about the system. User
  experience is dynamic, because it changes
  over time as the circumstances change.
Ok… So what is UX Benchmarking?
Is UX Benchmarking…




                      Not really
What is UX Benchmarking?


 The process of comparing a website’s (or any
  digital product) performance, specifically in
  relation to UX metrics, including among
  others effectiveness, efficiency, overall
  satisfaction and quality perception, to another
  that is widely considered to be an industry
  standard benchmark or best practice

 UX Benchmarking provides a snapshot of the
  performance of your product and helps you
  understand where you are in relation to a
  particular standard.
The question we’d like to address today is:




 How do you conduct UX Benchmarking?
In order to Benchmark, first you need to measure
How to Conduct UX Benchmarking
Why is UX Benchmarking important?

 Measuring the quality of UX is more important than ever

 Consumers have gained a lot of power in recent years

 More competition, more maturity in the online marketplace

 You can’t really manage what you can’t measure. UX Pros must
  deliver more measured value. Mgmt Teams LOVE metrics…

 UX Benchmarking allows you to MEASURE performance in two
  ways:

   1.   Baseline studies: Measure UX over time, compare evolution of your
        own product and/or service

   2.   Against competitors within your industry
Common UX research & design challenges


 Difficulties obtaining objective, statistically significant, actionable
  data

 Obtaining idea adoption from stakeholders

 Geographic/demographic limitations

 UX research is costly and time-consuming

 UX Benchmarking is therefore quite challenging

     This is where large-scale, remote unmoderated
                usability testing really helps
3 pillars of UX Benchmarking


                              1. Plan &
                                Goals




                          UX Benchmarking



                                            2. UX research
          3. Recruiting
          participants
                                              methods &
                                                 tools
4 must-dos with UX Benchmarking

1.   Plan ahead:

        Define measurable & comparable UX-related KPIs (will show them in the case study ahead)

        Select competitors to benchmark against

        Define target audience

        Identify common site activities/tasks (recommended between-subjects design)

2.   Conduct research using task-based testing (see methods and tools later on)

3.   Gather both quantitative & qualitative data

        Use the quantitative data to test for statistically significant differences between

         means or proportions

        Use qualitative data to support your findings and understand the why

4.   Research at least twice a year (or even quarterly) to measure design changes

     and trends in expectations
Define measurable & comparable UX-related KPIs




                                             Jeff Sauro,
                                              Measuringusability.com
Define measurable & comparable UX-related KPIs




                                                Jeff Sauro,
                                                 Measuringusability.com
UX research methods & tools

We recommend the following combination:


• Remote Unmoderated Usability Testing (or task-based survey tools)




• Remote Unmoderated Usability Testing Video Tools
What about recruiting participants?




     3 choices to recruit participants:
       1. Panel vendor (SSI, Research Now, etc)
       2. Private Mailing List
       3. Intercept visitors
Case study:
Benchmark Study to compare AA.com vs Delta.com
Study Background

                              • Competitive benchmark of American Airlines vs. Delta

                   Study
                                Air Lines websites
                              • Using task-based surveys, or Remote Unmoderated
                                Usability Testing
                              • Date: April 24 – April 25, 2012



                              • 100 participants (50 per site)

                   Panel
                              • Must fly for business or leisure more than twice a
                                year
                              • Must use websites to make travel arrangements
                              • Panel source: Survey Sampling International (SSI)




                   Tasks
                              • Go through the entire process of booking a flight
                              • Locate the total price for checking in two bags




www.userzoom.com
Task-based Surveys
                                                 (aka unmoderated large-scale remote usability testing)




           Go through the entire process of booking a flight




                   Go through the entire process of booking a flight




                                                                  • Large sample of participants take the study simultaneously
                                                                                                             • Natural context
                                                                                              • International/domestic testing
                                                                                                     • No moderation needed
                                                                                            • Usability + Behavior + Feedback

www.userzoom.com
Study Design



                                        American
                                        Airlines    Task 1   Task 2



  Screener         Pre-task questions
                                                                      Follow-up q’s


                                         Delta
                                        Air Lines   Task 1   Task 2




www.userzoom.com
Study Design

                               Pre-task brand attributes: Which of the following words do
           Pre-task            you associate with the AIRLINE brand?




                               Go through the entire process of booking a flight
              Tasks            Locate the total price for checking in two bags



                               Self-reported success
                               How satisfied were you with the process of booking a flight?
                               Ease of Use
         Post-tasks            Ease of navigation
                               Clear where to start
                               Satisfaction with amount of time to complete task


                                Overall rating
                                Post-task brand attributes
           Wrap-up              What did you like the most about the AIRLINE website
                                What did you like the least about the AIRLINE website
                                Likelihood to recommend (Net Promoter Score )

www.userzoom.com
Participant Profile

                         Age range                                                  Gender
                                                Under 18
                   14%                          18-20
             8%              31%                21-30
                                                                                            45%            Male
                                                31-40
                                                                        55%                                Female
                                                41-50
              27%
                           20%                  51-60
                                                60+



                                 How often do you fly yearly (for business or leisure)?
                             I don't fly regularly

                                       10+ times
                                                                                                  Participants were
                                      5-10 times                                                  required to book
                                                                                                  their travel online in
                                       2-4 times                                                  order to qualify

                                          1 time

                                                     0%    20%    40%         60%     80%

www.userzoom.com
Brand Perception




                                                 %
                                 Pre    Post   Change    Pre   Post   % Change


                   Friendly      41%    14%      -27%    42%   10%       -32%



                   Customer      27%    27%     0%       26%   40%       +14%
                    Focused


                     Safe        57%    27%       -30%   44%   24%       -20%



                    Good         41%    31%      -10%    36%   22%       -14%
                    value


www.userzoom.com
Task 1:
                         Finding a round-trip flight
Your Task: Find a round-trip flight departing May 1st and returning May 6th that you would consider booking on Delta Air
Lines/American Airlines website. You are travelling from San Francisco, CA to New York, NY. After that, please go through
the entire process of booking a flight until you get to the "Purchase" button.



       Delta Air Lines                                                 American Airlines
               Task Success                                                    Task Success




     Median Success Time                                              Median Success Time
            3:43                                                             3:56
On both studies the majority of participants self-reported being able to successfully complete the
booking process. There was not a significant difference in success rate or completion time (90%
Confidence Level).

www.userzoom.com
Task 1:
                          Finding a round-trip flight


Post-Task Questionnaire
Top Box Scores*

How satisfied were you with the process
of booking the flight?                                              50%                                  55%

Booking the flight was easy
                                                                    64%                                  67%

It was easy to navigate the website
                                                                    64%                                  65%

It was clear how to start booking the
flight                                                              62%                                  82%

I am satisfied with the amount of time it
took to complete the task                                           64%                                  80%

Participants that completed the American Airlines task reported being clearer on where to start and
more satisfied with the time it took to complete booking (90% Confidence Level). All percentages
represent top box scores.

*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
www.userzoom.com
Task 2:
                      Find the price for checking 2 bags
Your Task: Find the total price for checking in two bags for your flight from San Francisco to New York.
You purchased a standard economy class ticket.



       Delta Air Lines                                        American Airlines
               Task Success                                          Task Success




     Median Success Time                                     Median Success Time
            1:48                                                    2:00
On both studies the majority of participants were not able to successfully identify the price of
checking two bags. There was not a significant difference in success rate or completion time (90%
Confidence Level).

www.userzoom.com
Task 2:
                     Heatmaps - First Click




On both Delta and American Airlines participants used similar strategies when searching for baggage
checking fees. The most commonly used strategy was to start by clicking the flight booking form
fields/buttons (28% Delta “To Airport” field and 24% American Airlines “Find Flights” button).

Participants that first located and clicked on a “Baggage” had higher success rates (70% Delta and 60%
American Airlines).

www.userzoom.com
Task 2:
                          Find the price for checking 2 bags


 Post-Task Questionnaire
 Top Box Scores*

How satisfied were you with the process
of searching for the total price for                                30%                                  20%
checking in two bags?
 Finding the total price was easy
                                                                    32%                                  24%

It was easy to navigate the website
                                                                    38%                                  36%

It was clear how to start searching for
the total price                                                     32%                                  32%

I am satisfied with the amount of time it
took to complete the task                                           34%                                  34%

Participants that completed the Delta task reported being more satisfied with the process and ease of
the task. The differences are not significant (90% Confidence Level). All percentages represent top
box scores.
*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
 www.userzoom.com
Final Questionnaire:
                       What did you like the most about AA/Delta?




Representative Quotes:                                   Representative Quotes:

“very easy to navigate”                                  “the prices”
“easy to find flights”                                   “easy to navigate”
“The modern look of it”


Several of the Delta participants commented that they liked the appearance of the Delta website.
There were multiple American Airlines participants that responded liking American Airline’s pricing.

www.userzoom.com
Final Questionnaire:
                       What did you like the least about AA/Delta?




Representative Quotes:                               Representative Quotes:

“the pricing”                                        “finding information regarding
“Not easy to find bag fees”                          baggage pricing”
                                                     “Prices”
                                                     “finding things other than straight
                                                     flight booking”
For both websites participants commented that the pricing was high and that it was difficult to locate
the bag fees.

www.userzoom.com
Final Questionnaire:
                         Net Promoter Score


                   NPS           =    % of Promoters
                                       (9s and 10s)       -      Detractors
                                                               (0 through 6)




                             0   1      2     3    4     5     6    7    8      9    10    NPS

                            0%   4%    4%    4%   6%     18%   6%   10% 22%     6%   20% -16%
       Delta Air Lines
                            4%   6%    4%    0%   4%     10% 12% 18% 14% 12% 18% -10%
       American Airlines



                                            Detractors                         Promoters
            When asked if they would recommend the Delta/AA website to a family member,
            friend, or colleague both sites had negative NPS scores.


www.userzoom.com
Some takeaways…


1. UX Benchmarking, while challenging, is not mission impossible

2. Given the right plan, methods and tools, results are really valuable

3. UX Pros must deliver more measured value

4. There are plenty of UX metrics to be measured, including:
      Task success

      Time on task

      NPS

      Reasons for problems, abandonment

      The WHYs

      Etc.
Q&A
Thanks so much for your time!

                              Check out our upcoming news and events at www.userzoom.com




Sunnyvale (USA)                Barcelona (Spain)              Cheshire (UK)

440 N. Wolfe Rd. Sunnyvale,    Av. Diagonal 419 3º 2ª 08008   Booths Park. Chelford Road, Knutsford,
CA 94085                       Barcelona Phone: +34 93 414    Cheshire WA16 8GS
Phone: +1 (408) 524 7445       7554 Fax: +34 93 209 8380      Phone: + 44 (0)1565 759890
Contact: Alfonso de la Nuez    Contact: Xavier Mestres        Cell: + 44 (0)7900 472 920
                                                              Contact: Arthur Moan

More Related Content

What's hot

Moving the Needle with UX Benchmarking
Moving the Needle with UX BenchmarkingMoving the Needle with UX Benchmarking
Moving the Needle with UX BenchmarkingUserZoom
 
The HEART framework for UX metrics
The HEART framework for UX metricsThe HEART framework for UX metrics
The HEART framework for UX metricsUX Nights
 
What is UX, in 10 Slides
What is UX, in 10 SlidesWhat is UX, in 10 Slides
What is UX, in 10 SlidesJordan Julien
 
Simple Steps to UX/UI Web Design
Simple Steps to UX/UI Web DesignSimple Steps to UX/UI Web Design
Simple Steps to UX/UI Web DesignKoombea
 
UX STRAT Europe 2021 Workshop: Jules Skopp, Expedia
UX STRAT Europe 2021 Workshop: Jules Skopp, ExpediaUX STRAT Europe 2021 Workshop: Jules Skopp, Expedia
UX STRAT Europe 2021 Workshop: Jules Skopp, ExpediaUX STRAT
 
An Introduction to Usability Testing
An Introduction to Usability TestingAn Introduction to Usability Testing
An Introduction to Usability TestingLennart Overkamp
 
UI vs UX: Comparison Between User Interface and User Experience
UI vs UX: Comparison Between User Interface and User ExperienceUI vs UX: Comparison Between User Interface and User Experience
UI vs UX: Comparison Between User Interface and User ExperienceCloud Analogy
 
UX & UI Design - Differentiate through design
UX & UI Design - Differentiate through designUX & UI Design - Differentiate through design
UX & UI Design - Differentiate through designDMI
 
Google HEART UX metrics framework
Google HEART UX metrics framework Google HEART UX metrics framework
Google HEART UX metrics framework Raed Marji
 
Putting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for StrategyPutting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for StrategyUsability Matters
 
User Research Interviews: All you need to know
User Research Interviews: All you need to knowUser Research Interviews: All you need to know
User Research Interviews: All you need to knowNorthern User Experience
 
Prototyping for effective UX
Prototyping for effective UXPrototyping for effective UX
Prototyping for effective UXRan Liron
 
UI/UX Workshop - Hackvision
UI/UX Workshop - HackvisionUI/UX Workshop - Hackvision
UI/UX Workshop - HackvisionProttay Karim
 
UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied   UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied UXPA International
 
UX Strategy Blueprint
UX Strategy BlueprintUX Strategy Blueprint
UX Strategy BlueprintJim Kalbach
 

What's hot (20)

Moving the Needle with UX Benchmarking
Moving the Needle with UX BenchmarkingMoving the Needle with UX Benchmarking
Moving the Needle with UX Benchmarking
 
The HEART framework for UX metrics
The HEART framework for UX metricsThe HEART framework for UX metrics
The HEART framework for UX metrics
 
Ab testing
Ab testingAb testing
Ab testing
 
What is UX, in 10 Slides
What is UX, in 10 SlidesWhat is UX, in 10 Slides
What is UX, in 10 Slides
 
Simple Steps to UX/UI Web Design
Simple Steps to UX/UI Web DesignSimple Steps to UX/UI Web Design
Simple Steps to UX/UI Web Design
 
UX STRAT Europe 2021 Workshop: Jules Skopp, Expedia
UX STRAT Europe 2021 Workshop: Jules Skopp, ExpediaUX STRAT Europe 2021 Workshop: Jules Skopp, Expedia
UX STRAT Europe 2021 Workshop: Jules Skopp, Expedia
 
What is UX?
What is UX?What is UX?
What is UX?
 
An Introduction to Usability Testing
An Introduction to Usability TestingAn Introduction to Usability Testing
An Introduction to Usability Testing
 
Usability testing
Usability testing  Usability testing
Usability testing
 
UI vs UX: Comparison Between User Interface and User Experience
UI vs UX: Comparison Between User Interface and User ExperienceUI vs UX: Comparison Between User Interface and User Experience
UI vs UX: Comparison Between User Interface and User Experience
 
Usability Testing
Usability TestingUsability Testing
Usability Testing
 
UX & UI Design - Differentiate through design
UX & UI Design - Differentiate through designUX & UI Design - Differentiate through design
UX & UI Design - Differentiate through design
 
Google HEART UX metrics framework
Google HEART UX metrics framework Google HEART UX metrics framework
Google HEART UX metrics framework
 
UX/UI Introduction
UX/UI IntroductionUX/UI Introduction
UX/UI Introduction
 
Putting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for StrategyPutting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for Strategy
 
User Research Interviews: All you need to know
User Research Interviews: All you need to knowUser Research Interviews: All you need to know
User Research Interviews: All you need to know
 
Prototyping for effective UX
Prototyping for effective UXPrototyping for effective UX
Prototyping for effective UX
 
UI/UX Workshop - Hackvision
UI/UX Workshop - HackvisionUI/UX Workshop - Hackvision
UI/UX Workshop - Hackvision
 
UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied   UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied
 
UX Strategy Blueprint
UX Strategy BlueprintUX Strategy Blueprint
UX Strategy Blueprint
 

Viewers also liked

How to Benchmark Your Online Customer Experience Against Competition
How to Benchmark Your Online Customer Experience Against CompetitionHow to Benchmark Your Online Customer Experience Against Competition
How to Benchmark Your Online Customer Experience Against CompetitionUserZoom
 
Benchmark sites agences interim
Benchmark sites agences interimBenchmark sites agences interim
Benchmark sites agences interimBioforce
 
Benchmarking
BenchmarkingBenchmarking
BenchmarkingOmar BE
 
Benchmarking e-commerce en 3 concepts et 12 sites marchands
Benchmarking e-commerce en 3 concepts et 12 sites marchandsBenchmarking e-commerce en 3 concepts et 12 sites marchands
Benchmarking e-commerce en 3 concepts et 12 sites marchandsBenjamin Thiers
 
Rapport supref benchmarking 2016
Rapport supref benchmarking 2016Rapport supref benchmarking 2016
Rapport supref benchmarking 2016Amaury Baot
 
Benchmark Sites Agences Com. Sensible
Benchmark Sites Agences Com. SensibleBenchmark Sites Agences Com. Sensible
Benchmark Sites Agences Com. SensibleBioforce
 
Benchmarking ppt
Benchmarking pptBenchmarking ppt
Benchmarking pptAMARAYYA
 

Viewers also liked (8)

How to Benchmark Your Online Customer Experience Against Competition
How to Benchmark Your Online Customer Experience Against CompetitionHow to Benchmark Your Online Customer Experience Against Competition
How to Benchmark Your Online Customer Experience Against Competition
 
Benchmark sites agences interim
Benchmark sites agences interimBenchmark sites agences interim
Benchmark sites agences interim
 
Benchmarking
BenchmarkingBenchmarking
Benchmarking
 
Benchmarking e-commerce en 3 concepts et 12 sites marchands
Benchmarking e-commerce en 3 concepts et 12 sites marchandsBenchmarking e-commerce en 3 concepts et 12 sites marchands
Benchmarking e-commerce en 3 concepts et 12 sites marchands
 
Rapport supref benchmarking 2016
Rapport supref benchmarking 2016Rapport supref benchmarking 2016
Rapport supref benchmarking 2016
 
Benchmark Sites Agences Com. Sensible
Benchmark Sites Agences Com. SensibleBenchmark Sites Agences Com. Sensible
Benchmark Sites Agences Com. Sensible
 
Benchmark concurrence
Benchmark concurrenceBenchmark concurrence
Benchmark concurrence
 
Benchmarking ppt
Benchmarking pptBenchmarking ppt
Benchmarking ppt
 

Similar to How to Conduct UX Benchmarking

Userzoom Webinar Monster Aug09
Userzoom Webinar Monster Aug09Userzoom Webinar Monster Aug09
Userzoom Webinar Monster Aug09Alfonso de la Nuez
 
UserZoom Big Design Conf May09
UserZoom Big Design Conf May09UserZoom Big Design Conf May09
UserZoom Big Design Conf May09Alfonso de la Nuez
 
User Zoom Webinar Monster Aug09
User Zoom Webinar Monster Aug09User Zoom Webinar Monster Aug09
User Zoom Webinar Monster Aug09guest07f4705
 
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...Carine Lallemand
 
UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking
UserZoom Webinar: How to Conduct Web Customer Experience BenchmarkingUserZoom Webinar: How to Conduct Web Customer Experience Benchmarking
UserZoom Webinar: How to Conduct Web Customer Experience BenchmarkingUserZoom
 
User-Testing, Testing, 1,2,3
User-Testing, Testing, 1,2,3User-Testing, Testing, 1,2,3
User-Testing, Testing, 1,2,3BusinessOnline
 
Remote Testing Methods & Tools Webinar
Remote Testing Methods & Tools WebinarRemote Testing Methods & Tools Webinar
Remote Testing Methods & Tools WebinarUserZoom
 
Thomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfThomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfQA or the Highway
 
Webinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability TestingWebinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability TestingUserZoom
 
Uz big design talk may10
Uz big design talk may10Uz big design talk may10
Uz big design talk may10UserZoom
 
Usability testing for qualitative researchers
Usability testing for qualitative researchersUsability testing for qualitative researchers
Usability testing for qualitative researchersResearchShare
 
Usability testing for qualitative researchers
Usability testing for qualitative researchersUsability testing for qualitative researchers
Usability testing for qualitative researchersKay Corry Aubrey
 
Web Usability (Slideshare Version)
Web Usability (Slideshare Version)Web Usability (Slideshare Version)
Web Usability (Slideshare Version)Carles Farré
 
Workshop: Managing top tasks #BPCW11
Workshop: Managing top tasks #BPCW11Workshop: Managing top tasks #BPCW11
Workshop: Managing top tasks #BPCW11Headstar
 
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo EuteneuerStop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo EuteneuerJAXLondon_Conference
 
Software Testing Introduction (Part 4))
 Software Testing Introduction (Part 4)) Software Testing Introduction (Part 4))
Software Testing Introduction (Part 4))Thapar Institute
 
UXprobe workshop at Dare Festival 2016
UXprobe workshop at Dare Festival 2016UXprobe workshop at Dare Festival 2016
UXprobe workshop at Dare Festival 2016UXprobe
 

Similar to How to Conduct UX Benchmarking (20)

Userzoom Webinar Monster Aug09
Userzoom Webinar Monster Aug09Userzoom Webinar Monster Aug09
Userzoom Webinar Monster Aug09
 
UserZoom Big Design Conf May09
UserZoom Big Design Conf May09UserZoom Big Design Conf May09
UserZoom Big Design Conf May09
 
User Zoom Webinar Monster Aug09
User Zoom Webinar Monster Aug09User Zoom Webinar Monster Aug09
User Zoom Webinar Monster Aug09
 
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...
NORDICHI'14 - Carine Lallemand - How Relevant is an Expert Evaluation of UX b...
 
UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking
UserZoom Webinar: How to Conduct Web Customer Experience BenchmarkingUserZoom Webinar: How to Conduct Web Customer Experience Benchmarking
UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking
 
User-Testing, Testing, 1,2,3
User-Testing, Testing, 1,2,3User-Testing, Testing, 1,2,3
User-Testing, Testing, 1,2,3
 
Remote Testing Methods & Tools Webinar
Remote Testing Methods & Tools WebinarRemote Testing Methods & Tools Webinar
Remote Testing Methods & Tools Webinar
 
Thomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfThomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdf
 
Webinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability TestingWebinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability Testing
 
Usability_Evaluation
Usability_EvaluationUsability_Evaluation
Usability_Evaluation
 
Uz big design talk may10
Uz big design talk may10Uz big design talk may10
Uz big design talk may10
 
Usability testing for qualitative researchers
Usability testing for qualitative researchersUsability testing for qualitative researchers
Usability testing for qualitative researchers
 
Usability testing for qualitative researchers
Usability testing for qualitative researchersUsability testing for qualitative researchers
Usability testing for qualitative researchers
 
Web Usability (Slideshare Version)
Web Usability (Slideshare Version)Web Usability (Slideshare Version)
Web Usability (Slideshare Version)
 
Workshop: Managing top tasks #BPCW11
Workshop: Managing top tasks #BPCW11Workshop: Managing top tasks #BPCW11
Workshop: Managing top tasks #BPCW11
 
Levi McCusker UXD
Levi McCusker UXDLevi McCusker UXD
Levi McCusker UXD
 
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo EuteneuerStop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
 
Software Testing Introduction (Part 4))
 Software Testing Introduction (Part 4)) Software Testing Introduction (Part 4))
Software Testing Introduction (Part 4))
 
UXprobe workshop at Dare Festival 2016
UXprobe workshop at Dare Festival 2016UXprobe workshop at Dare Festival 2016
UXprobe workshop at Dare Festival 2016
 
Unit 09: Web Application Testing
Unit 09: Web Application TestingUnit 09: Web Application Testing
Unit 09: Web Application Testing
 

More from UserZoom

Retail UX in 2020: How to stay on top of changing customer behaviors
Retail UX in 2020: How to stay on top of changing customer behaviorsRetail UX in 2020: How to stay on top of changing customer behaviors
Retail UX in 2020: How to stay on top of changing customer behaviorsUserZoom
 
[Webinar] Transitioning to Remote Research
[Webinar] Transitioning to Remote Research[Webinar] Transitioning to Remote Research
[Webinar] Transitioning to Remote ResearchUserZoom
 
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...UserZoom
 
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...UserZoom
 
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with ConfidenceUserZoom
 
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...UserZoom
 
How to Work Collaboratively on the User Experience
How to Work Collaboratively on the User ExperienceHow to Work Collaboratively on the User Experience
How to Work Collaboratively on the User ExperienceUserZoom
 
UX Research for the Masses
UX Research for the MassesUX Research for the Masses
UX Research for the MassesUserZoom
 
Don’t Guess It, Test It!
Don’t Guess It, Test It!Don’t Guess It, Test It!
Don’t Guess It, Test It!UserZoom
 
Sharing the Love: Democratising Research at MoneySuperMarket
Sharing the Love: Democratising Research at MoneySuperMarketSharing the Love: Democratising Research at MoneySuperMarket
Sharing the Love: Democratising Research at MoneySuperMarketUserZoom
 
Making Research a UX Team Sport
Making Research a UX Team SportMaking Research a UX Team Sport
Making Research a UX Team SportUserZoom
 
How to Help Newbies Run UX Tests
How to Help Newbies Run UX TestsHow to Help Newbies Run UX Tests
How to Help Newbies Run UX TestsUserZoom
 
Ready to Democratise UX Insights?
Ready to Democratise UX Insights?Ready to Democratise UX Insights?
Ready to Democratise UX Insights?UserZoom
 
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...UserZoom
 
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX Insights
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX InsightsHow to Master UserZoom’s Latest Study Builder & Extract Meaningful UX Insights
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX InsightsUserZoom
 
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...UserZoom
 
How to Run Research in Agile Sprints by Democratizing It Across Teams
How to Run Research in Agile Sprints by Democratizing It Across TeamsHow to Run Research in Agile Sprints by Democratizing It Across Teams
How to Run Research in Agile Sprints by Democratizing It Across TeamsUserZoom
 
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...UserZoom
 
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...UserZoom
 
Telling Your UX Metrics Story - The 21st Century Metrics Model
Telling Your UX Metrics Story - The 21st Century Metrics ModelTelling Your UX Metrics Story - The 21st Century Metrics Model
Telling Your UX Metrics Story - The 21st Century Metrics ModelUserZoom
 

More from UserZoom (20)

Retail UX in 2020: How to stay on top of changing customer behaviors
Retail UX in 2020: How to stay on top of changing customer behaviorsRetail UX in 2020: How to stay on top of changing customer behaviors
Retail UX in 2020: How to stay on top of changing customer behaviors
 
[Webinar] Transitioning to Remote Research
[Webinar] Transitioning to Remote Research[Webinar] Transitioning to Remote Research
[Webinar] Transitioning to Remote Research
 
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...
Moderated vs Unmoderated Research: It’s time to say ELMO (Enough, let’s move ...
 
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
 
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence
[19.2 UserZoom Spring Release Webinar] Get Card Sort Insights with Confidence
 
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...
Benchmarking Mini-series Part #2: Conducting Quick, Cost-Effective UX Benchma...
 
How to Work Collaboratively on the User Experience
How to Work Collaboratively on the User ExperienceHow to Work Collaboratively on the User Experience
How to Work Collaboratively on the User Experience
 
UX Research for the Masses
UX Research for the MassesUX Research for the Masses
UX Research for the Masses
 
Don’t Guess It, Test It!
Don’t Guess It, Test It!Don’t Guess It, Test It!
Don’t Guess It, Test It!
 
Sharing the Love: Democratising Research at MoneySuperMarket
Sharing the Love: Democratising Research at MoneySuperMarketSharing the Love: Democratising Research at MoneySuperMarket
Sharing the Love: Democratising Research at MoneySuperMarket
 
Making Research a UX Team Sport
Making Research a UX Team SportMaking Research a UX Team Sport
Making Research a UX Team Sport
 
How to Help Newbies Run UX Tests
How to Help Newbies Run UX TestsHow to Help Newbies Run UX Tests
How to Help Newbies Run UX Tests
 
Ready to Democratise UX Insights?
Ready to Democratise UX Insights?Ready to Democratise UX Insights?
Ready to Democratise UX Insights?
 
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...
BENCHMARKING MINI-SERIES PART #1: Proving Value & Quantifying the Impact of U...
 
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX Insights
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX InsightsHow to Master UserZoom’s Latest Study Builder & Extract Meaningful UX Insights
How to Master UserZoom’s Latest Study Builder & Extract Meaningful UX Insights
 
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
Exploring UX in the Enterprise: The Industry’s Hottest Trends & Insights from...
 
How to Run Research in Agile Sprints by Democratizing It Across Teams
How to Run Research in Agile Sprints by Democratizing It Across TeamsHow to Run Research in Agile Sprints by Democratizing It Across Teams
How to Run Research in Agile Sprints by Democratizing It Across Teams
 
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...
[Product Release Highlight] Seamlessly Discover and Share Your User Experienc...
 
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...
Hold on to Your Hats: The Scaled Agile Framework (SAFe) Might Actually Be a G...
 
Telling Your UX Metrics Story - The 21st Century Metrics Model
Telling Your UX Metrics Story - The 21st Century Metrics ModelTelling Your UX Metrics Story - The 21st Century Metrics Model
Telling Your UX Metrics Story - The 21st Century Metrics Model
 

Recently uploaded

Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 

Recently uploaded (20)

Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 

How to Conduct UX Benchmarking

  • 1. Webinar: How to Conduct UX Benchmarking May 2nd, 2012
  • 2. Intros Alfonso de la Nuez Kim Oslob Co-Founder & Co-CEO Sr. Client Services Director Twitter Hashtag #uzwebinar
  • 3. Overview  What we mean by UX Benchmarking and why it’s important?  How to conduct UX Benchmarking studies, using remote unmoderated usability testing (aka task-based surveys)?  What are some of the UX metrics you should look for?  How to measure the quality of a website’s user experience across various competitors?  A case study: American Airlines vs Delta Airlines Websites
  • 4. What is Benchmarking? Definition of Benchmarking:  The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice.  It provides a snapshot of the performance of your business and helps you understand where you are in relation to a particular standard.
  • 5. What is UX? Definition of UX:  It’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system.  User experience is subjective in nature, as it is about an individual’s feelings and thoughts about the system. User experience is dynamic, because it changes over time as the circumstances change.
  • 6. Ok… So what is UX Benchmarking?
  • 8. What is UX Benchmarking?  The process of comparing a website’s (or any digital product) performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception, to another that is widely considered to be an industry standard benchmark or best practice  UX Benchmarking provides a snapshot of the performance of your product and helps you understand where you are in relation to a particular standard.
  • 9. The question we’d like to address today is:  How do you conduct UX Benchmarking?
  • 10. In order to Benchmark, first you need to measure
  • 12. Why is UX Benchmarking important?  Measuring the quality of UX is more important than ever  Consumers have gained a lot of power in recent years  More competition, more maturity in the online marketplace  You can’t really manage what you can’t measure. UX Pros must deliver more measured value. Mgmt Teams LOVE metrics…  UX Benchmarking allows you to MEASURE performance in two ways: 1. Baseline studies: Measure UX over time, compare evolution of your own product and/or service 2. Against competitors within your industry
  • 13. Common UX research & design challenges  Difficulties obtaining objective, statistically significant, actionable data  Obtaining idea adoption from stakeholders  Geographic/demographic limitations  UX research is costly and time-consuming  UX Benchmarking is therefore quite challenging This is where large-scale, remote unmoderated usability testing really helps
  • 14. 3 pillars of UX Benchmarking 1. Plan & Goals UX Benchmarking 2. UX research 3. Recruiting participants methods & tools
  • 15. 4 must-dos with UX Benchmarking 1. Plan ahead:  Define measurable & comparable UX-related KPIs (will show them in the case study ahead)  Select competitors to benchmark against  Define target audience  Identify common site activities/tasks (recommended between-subjects design) 2. Conduct research using task-based testing (see methods and tools later on) 3. Gather both quantitative & qualitative data  Use the quantitative data to test for statistically significant differences between means or proportions  Use qualitative data to support your findings and understand the why 4. Research at least twice a year (or even quarterly) to measure design changes and trends in expectations
  • 16. Define measurable & comparable UX-related KPIs  Jeff Sauro, Measuringusability.com
  • 17. Define measurable & comparable UX-related KPIs  Jeff Sauro, Measuringusability.com
  • 18. UX research methods & tools We recommend the following combination: • Remote Unmoderated Usability Testing (or task-based survey tools) • Remote Unmoderated Usability Testing Video Tools
  • 19. What about recruiting participants?  3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors
  • 20. Case study: Benchmark Study to compare AA.com vs Delta.com
  • 21. Study Background • Competitive benchmark of American Airlines vs. Delta Study Air Lines websites • Using task-based surveys, or Remote Unmoderated Usability Testing • Date: April 24 – April 25, 2012 • 100 participants (50 per site) Panel • Must fly for business or leisure more than twice a year • Must use websites to make travel arrangements • Panel source: Survey Sampling International (SSI) Tasks • Go through the entire process of booking a flight • Locate the total price for checking in two bags www.userzoom.com
  • 22. Task-based Surveys (aka unmoderated large-scale remote usability testing) Go through the entire process of booking a flight Go through the entire process of booking a flight • Large sample of participants take the study simultaneously • Natural context • International/domestic testing • No moderation needed • Usability + Behavior + Feedback www.userzoom.com
  • 23. Study Design American Airlines Task 1 Task 2 Screener Pre-task questions Follow-up q’s Delta Air Lines Task 1 Task 2 www.userzoom.com
  • 24. Study Design Pre-task brand attributes: Which of the following words do Pre-task you associate with the AIRLINE brand? Go through the entire process of booking a flight Tasks Locate the total price for checking in two bags Self-reported success How satisfied were you with the process of booking a flight? Ease of Use Post-tasks Ease of navigation Clear where to start Satisfaction with amount of time to complete task Overall rating Post-task brand attributes Wrap-up What did you like the most about the AIRLINE website What did you like the least about the AIRLINE website Likelihood to recommend (Net Promoter Score ) www.userzoom.com
  • 25. Participant Profile Age range Gender Under 18 14% 18-20 8% 31% 21-30 45% Male 31-40 55% Female 41-50 27% 20% 51-60 60+ How often do you fly yearly (for business or leisure)? I don't fly regularly 10+ times Participants were 5-10 times required to book their travel online in 2-4 times order to qualify 1 time 0% 20% 40% 60% 80% www.userzoom.com
  • 26. Brand Perception % Pre Post Change Pre Post % Change Friendly 41% 14% -27% 42% 10% -32% Customer 27% 27% 0% 26% 40% +14% Focused Safe 57% 27% -30% 44% 24% -20% Good 41% 31% -10% 36% 22% -14% value www.userzoom.com
  • 27. Task 1: Finding a round-trip flight Your Task: Find a round-trip flight departing May 1st and returning May 6th that you would consider booking on Delta Air Lines/American Airlines website. You are travelling from San Francisco, CA to New York, NY. After that, please go through the entire process of booking a flight until you get to the "Purchase" button. Delta Air Lines American Airlines Task Success Task Success Median Success Time Median Success Time 3:43 3:56 On both studies the majority of participants self-reported being able to successfully complete the booking process. There was not a significant difference in success rate or completion time (90% Confidence Level). www.userzoom.com
  • 28. Task 1: Finding a round-trip flight Post-Task Questionnaire Top Box Scores* How satisfied were you with the process of booking the flight? 50% 55% Booking the flight was easy 64% 67% It was easy to navigate the website 64% 65% It was clear how to start booking the flight 62% 82% I am satisfied with the amount of time it took to complete the task 64% 80% Participants that completed the American Airlines task reported being clearer on where to start and more satisfied with the time it took to complete booking (90% Confidence Level). All percentages represent top box scores. *Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale) www.userzoom.com
  • 29. Task 2: Find the price for checking 2 bags Your Task: Find the total price for checking in two bags for your flight from San Francisco to New York. You purchased a standard economy class ticket. Delta Air Lines American Airlines Task Success Task Success Median Success Time Median Success Time 1:48 2:00 On both studies the majority of participants were not able to successfully identify the price of checking two bags. There was not a significant difference in success rate or completion time (90% Confidence Level). www.userzoom.com
  • 30. Task 2: Heatmaps - First Click On both Delta and American Airlines participants used similar strategies when searching for baggage checking fees. The most commonly used strategy was to start by clicking the flight booking form fields/buttons (28% Delta “To Airport” field and 24% American Airlines “Find Flights” button). Participants that first located and clicked on a “Baggage” had higher success rates (70% Delta and 60% American Airlines). www.userzoom.com
  • 31. Task 2: Find the price for checking 2 bags Post-Task Questionnaire Top Box Scores* How satisfied were you with the process of searching for the total price for 30% 20% checking in two bags? Finding the total price was easy 32% 24% It was easy to navigate the website 38% 36% It was clear how to start searching for the total price 32% 32% I am satisfied with the amount of time it took to complete the task 34% 34% Participants that completed the Delta task reported being more satisfied with the process and ease of the task. The differences are not significant (90% Confidence Level). All percentages represent top box scores. *Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale) www.userzoom.com
  • 32. Final Questionnaire: What did you like the most about AA/Delta? Representative Quotes: Representative Quotes: “very easy to navigate” “the prices” “easy to find flights” “easy to navigate” “The modern look of it” Several of the Delta participants commented that they liked the appearance of the Delta website. There were multiple American Airlines participants that responded liking American Airline’s pricing. www.userzoom.com
  • 33. Final Questionnaire: What did you like the least about AA/Delta? Representative Quotes: Representative Quotes: “the pricing” “finding information regarding “Not easy to find bag fees” baggage pricing” “Prices” “finding things other than straight flight booking” For both websites participants commented that the pricing was high and that it was difficult to locate the bag fees. www.userzoom.com
  • 34. Final Questionnaire: Net Promoter Score NPS = % of Promoters (9s and 10s) - Detractors (0 through 6) 0 1 2 3 4 5 6 7 8 9 10 NPS 0% 4% 4% 4% 6% 18% 6% 10% 22% 6% 20% -16% Delta Air Lines 4% 6% 4% 0% 4% 10% 12% 18% 14% 12% 18% -10% American Airlines Detractors Promoters When asked if they would recommend the Delta/AA website to a family member, friend, or colleague both sites had negative NPS scores. www.userzoom.com
  • 35. Some takeaways… 1. UX Benchmarking, while challenging, is not mission impossible 2. Given the right plan, methods and tools, results are really valuable 3. UX Pros must deliver more measured value 4. There are plenty of UX metrics to be measured, including:  Task success  Time on task  NPS  Reasons for problems, abandonment  The WHYs  Etc.
  • 36. Q&A
  • 37. Thanks so much for your time! Check out our upcoming news and events at www.userzoom.com Sunnyvale (USA) Barcelona (Spain) Cheshire (UK) 440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford, CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GS Phone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890 Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920 Contact: Arthur Moan

Editor's Notes

  1. In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  2. In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  3. In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  4. In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.