Presented 5/11/17 @LOCO_UX by @jkooda of @liminaUX This talk covers the anatomy of a UX Eval, how to use it as a business development tool, and how to ensure you have a logical and most importantly beneficial return on your client's investment.

1. The Power of
The UX Evaluation
a LOCO | UX Meetup
TONIGHT

2. A FEW SHORT WORDS
LOCO | UX Community Mission
Build a network of UX practitioners local to the Longmont/Boulder
area for the purpose of:
• Access to opportunities (seeking, hiring, colabos)
• Sharing Ideas, Tools, Methods, Techniques
• Building community for mutual enhancement/benefit
• each other
• local businesses
• Longmont municipality

3. Who Tweets During Meetups?
@jkooda of @liminaUX at @loco_ux

4. Expert
Heuristic (huh?)
Evaluation

5. heu·ris·tic hyo͞oˈristik/
adjective
enabling a person to discover or learn something for
themselves.
"a “hands-on” or interactive heuristic approach to learning"
noun
a heuristic process or method.

6. Is a usability engineering method for finding usability
problems in a user interface design, thereby making
them addressable and solvable.
It involves a small set of expert evaluators who examine
the interface and assess its compliance with
“heuristics,” or recognized usability principles.
Heuristic Evaluation

7. 10 Usability Heuristic Principles
Visibility of system status
Match between system and the
real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Helps users recognize, diagnose,
and recover from errors
Help and documentation
Examples in the Wild: http://designingwebinterfaces.com/6-tips-for-a-great-flex-ux-part-5
Source: https://www.nngroup.com/articles/ten-usability-heuristics/

8. Donald A. Norman
Jacob Nielson
“DISCOUNT
USABILITY
ENGINEERING
SERVICE”

9. CAVEAT
A heuristic evaluation should not
replace usability testing.
Although the heuristics relate to criteria that affect
your site’s usability, the issues identified in a heuristic
evaluation are different than those found in a
usability test.

10. PROS
• Provides quick and relatively inexpensive (cheap)
• Systematic and Intuitive, applying a set of predefined rules/heuristics
• Requires very little planning
• Can be used early in the design process
• Does not require a finished interface
• Provides corrective measures
• Can be used together with other usability testing methods
• Can help identify and prioritize issues in need of further usability
testing

11. CONS
• Requires knowledge and experience to apply the heuristics effectively
• Trained usability experts are sometimes hard to find and can be
expensive
• Requires multiple experts and aggregate their results
• The evaluation may identify more minor issues and fewer major issues
• Tends to focus on problems rather than solutions

12. 1. Set Up The Team
2. Agree on Evaluation Criteria
3. Conduct Cognitive Walk Through
4. Analyze Interface & Record Issues
5. Internal Review
6. Present to Client
7. … Next steps & business development
6 MAJOR PARTS

13. 1.WHO DO YOU NEED?

14. PICK 3
UX LEAD USER RESEARCHER INTERACTION DESIGNER
VISUAL DESIGNER DEVELOPER CONTENT SPECIALIST

15. WHY 3?

16. Rows = 19 evaluators
Column = 16 usability problems
= Problem found
Economics of the Evaluation Team
Source: https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
Weak Evaluator
Strong Evaluator
Hard Easy
0%
25%
50%
75%
100%
1 5 10 15
Number of Evaluators
PercentIssuesFound
Strength in Numbers
0%
10%
20%
30%
40%
50%
60%
70%
1 5 10 15
Number of Evaluators
Cost/BenefitRatio
ROI
Eval Quants

17. Freak-o-nomics
1.Emergence over authority – Talks about the shift in authority from the top-down to the bottom-up.
Think Wikipedia.
2.Pull over push – Another example of bottom-up demand. User-driven instead of mass-marketing.
Think on-demand instead of broadcast.
3.Compasses over maps – Methodology not mission. Teach a man to fish. A play for Seymour Papert’s
constructionism.
4.Risk over safety – When the cost of innovation is low, like it is now, it’s easier take more risks and try
more things. Negroponte famous MediaLab motto was, “Demo or Die”, Ito says “Deploy.”
5.Disobedience over compliance – “You don’t win a Nobel Prize by doing what you’re told” (Joi Ito). Ask
forgiveness, not permission.
6.Practice over theory – “In theory there is no difference between theory and practice. In practice there
is” (Yogi Berra). Both should inform one other.
7.Diversity over ability – Crowdsourcing (a word Howe coined) can solve
some problems that dedicated professions can’t solve internally.
8.Resilience over strength – They quote the old adage that an oak tree is strong, but can be blown over
in a storm; while the weaker but flexible willow bends with it and can survive. Akin to what Angela
Duckworth popularized as “grit” in her great book with the same title.
9.Systems over objects – The value of interdisciplinary networking.

18. 2. SO, WHO DOES WHAT?

19. • Workflow
• Information
Architecture
• Ease of Use
• Taxonomy /
Terminology
• Interface Elements
• Window/Pane
Management
• Error Handling
• Access to Help
• Page Layout
• Color Use
• Visual Language
• Data Visualization
• Iconography
BREAKDOWN OF RESPONSIBILITY
USER RESEARCHER INTERACTION DESIGNER VISUAL DESIGNER

20. 3. COGNITIVE WALKTHROUGH

21. Getting Domain Expertise
BUSINESS REPRESENTATIVES — Wrangle business objectives and understand
opportunities for the business. Understand how to make or save money for the
business. Understand how the design meets the needs of the product domain and its
users.
PROJECT MANAGER — Ensure the client can implement this design in time and
when/what the project milestones are.
TECHNICAL EXPERT — Understand technical framework (opportunities and limitations).
Validate the client can implement the product recommendations (feasibility / time
intensity).
YOUR TEAM - provides the domain expertise in “USER EXPERIENCE &
USABILITY”, they provide industry domain expertise.

22. SETTING TARGETS
KEY PRODUCT
FEATURES
Plan to evaluate all key
features that differentiate this
product from it’s competitors.
Ask who else is doing this, is
any one doing this better?
KEY TASKS /
WORKFLOWS
Plan to evaluate all key
workflows in the system.
Unless problematic, avoid
evaluating common tasks (ex:
login, logout)
KEY PERFORMANCE
INDICATORS
Understand what metrics are
important to the business:
(ex. time to completion, task
completion ratios, user
registration, engagement
volume, etc.)
BE A SPONGE:
• Take this opportunity to understand the product from the business and user perspective.
• Remain objective, do not adopt or project your cognitive bias.
• Take screenshots and where possible, record the session (voice and screen)

23. 4. GETCHO ANALYSIS ON!

24. Ingredients
SHARED REPOSITORY
• Directory: (Recording, Screenshots,
Documents, Tracking Sheets)
• Collaboration Workspace: Google Slides,
InvisionApp, 3rd Party Tools
EVALUATION STRUCTURE
• Outline by Issue Category or Functional
Areas
• Issue category overview
• Individual issue slides
• Category findings
SCREEN MARKUPS
• Point out problem areas
• Make note of successes
• Assign heuristic (1-13)
• Assign Issue Category
• Assign Severity / Prevalence (H, M, L)
• Cite example solutions and
recommendations
KEY TAKEAWAYS (Executive Summary)
• Findings
• Key Issues
• Severity/Priority

25. THANK YOU XEROX
• 22 Pages
• 13 Principles
• 288 Check Points
Source: http://users.polytech.unice.fr/~pinna/MODULEIHM/ANNEE2010/CEIHM/XEROX%20HE_CKLST.pdf

26. THANK YOU CHRIS GALLELLO
• Chrome Plugin
• 10 Principles
• - Heuristic
• - Severity
• - Notes
• - Recommendations
• In-Line Annotations
• Word.doc Export
Source: : http://www.uxcheck.co/
Reference: https://medium.com/@cgallello/low-cost-usability-testing-61b5f8a2a1be

27. 5. PRE-GAME HUDDLE

28. Draft Run Through
GAP ANALYSIS – Any areas need more
attention? Use your KPI’s, Key Workflows, Key
Product Features
INTERIM CLIENT CHECKPOINT – Depending
on confidence and success, it’s always a good
idea to check in with the client around the
60% complete mark
ORGANIZE FINDINGS — Look across what
everyone has analyzed, group logically (by
issue category or functional areas – avoid
redundancy (cite instances and
severity/prevalence)
DRIVE VALUE – Without giving up free work,
provide broad recommendations an cite
examples of successful implementations.
ASSIGN PRESENTERS / NOTE TAKERS —
Depending on skills mix you may want to mix
it up, unless you’re confident with one
presenter
ESTIMATE PRESENTATION TIME — These
reports can be dense and packed with rich
discussion, you will need to account for extra
time per issue.

29. 6. SHOW TIME

30. Empathy
BE SENSITIVE TO POLITICAL /
EMOTIONAL IMPLICATIONS
<VERY IMPORTANT>
• Always start on a positive note it’s important to
acknowledge where they hit the mark and
showed innovative, unique and successful
design
• Often times the interfaces you’re evaluating
represent a significant amount of commitment,
time, and/or cost investment on your client’s
part
• Realize that every observation can be perceived
as a personal critique by any number of client
team members
BREAK THE NEWS IN THE BEST LIGHT
POSSIBLE
• Your perspective represents an objective 3rd
party observations that should not be charged
with subjectivity and aesthetic/technical
preference
• This is particularly important for the visual
analysis
• Your observations should be supported by
empirical evidence and sound solutions and
recommendations
• Make room for conversations, check the pulse of
how each observation and recommendation is
landing

31. Engage
YOU MAY HAVE CLARIFYING QUESTIONS
• Make sure you’re taking time to validate your observations and recommendations
THIS IS NOT A PASSIVE SESSION
• Create opportunity for the client to articulate priority and effort levels against your
observations and recommendations (H, M, L) this will be useful in backlog
management
• You are proxy for the user, be sure to advocate for the usability impact of resolving or
deprioritizing a fix, they are relying on you to be the expert
• Continue to articulate your recommendations and solutions in terms of KPI’s and what
the would be most valuable to the business without negatively impacting ease of use
and user value.

32. Drive Momentum
IF YOU’VE DONE YOUR JOB…
• The client would have agreed with 60%-80% of your findings
• They will already be thinking about low handing fruit and how to parse the
enhancement into near term sprints and long term release cycles – this is your
opportunity to step up.
THE EVALUATION REPORT IS THE BEGINNING, NOT THE END
• You now own the issue log and recommendations with client feedback
• Ranked by: Client Priority, User Benefit, Technical Complexity…
• This is the making of what we call a Feature Matrix: a useful tool for UX
prioritization, sprint planning and backlog management

33. SAMPLE
UX EVAL
<ATTENDEES ONLY>

34. FEATUREMATIX
(sneakpeek)

35. • Catalogue UX
enhancements or
requirements into swim-
lanes
• Assign each cell with User
Benefit, Business Benefit
and Technical Complexity
• Use this information to build
the priority ranking for
sprint planning and backlog
management.

36. YOU SAID POWER.

37. POWER FOR YOU
YOUR BUSINESS DEVELOPMENT TOOL FOR UX WORK
Brings you top of mind for follow on detailed design contract work
Clearly articulates the value of UX in terms that matter to the client
Wets the appetite of the client without giving up detailed design work
Strong business planning tool – showing UX as a strategic practice

38. POWER FOR YOUR CLIENT
TACTICAL HITLIST
In most cases your observations and recommendations can dovetail
into their existing development plan – the client can yield immediate
value from the report
For a relatively low cost, the client has gained critical external
feedback for near and long term product enhancements – outputs
from the report are typically folded into long term development plans

39. THANK YOU!
Questions?

40. LOCO | UX
Community News?
Announcements?
Next Topic Ideas?

41. THANKS TO OUR SPONSORS!
https://twitter.com/loco_ux
+
CoSolve