You probably have noticed the new look of Atlassian's Cloud products. Our new Design Guidelines took many months to create, and our team had many tough decisions to make. Luckily, we incorporated customer research along the way to guide us. One of our most valuable research tools is called “early signal testing”, and we think it can help you too. Early signal testing can help you gain confidence in a direction, rather than being paralyzed by a choice. It can help assess your design's usability, clarity, comprehension, and more. This talk explains how your team can gather measurable user feedback in as little as a week, for even the very biggest of problems.

1. Early Signal Testing:
Designing Atlassian’s New Look
ASHLEIGH STERZENBACH | SENIOR DESIGNER | ATLASSIAN
BECKY WHITE | SENIOR RESEARCHER | ATLASSIAN

2. 35,000

8. Decisions can be dauntinglife-changingscary.exhaustingexcitingan upgrade

9. Product decisions can
be scary too.

16. We too made some scary
decisions recently.

17. N
View issue · Add to sprint 1.4-m1
Issue ADG-745 has been created.
MOVED
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
MOVED
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
123122
123122
123122
123122
123122
Alana AwareAlana Aware
Buttons
Forms
Tables
Tabs
Dropdown button Dropdown button
StatisticsReopenMoreCommentAssign to meAssignEdit
ConfirmInvite usersSelect pluginsConfigure
instance
Gray background

18. Open in JIRA Open in JIRAOpen in JIRA Open in JIRAOpen in JIRA Open in JIRA
RECENT PROJECTS
SELECTED
ALL PROJECTS
Issue rewrites
Issue Rewrite (IR)
JIRA Core Bord (JCB)
JIRA Core Board (JCB)
Vertigo Incompatibilties (VI)
Vertigo Incompatibilties (VI)
JIRA Process Advancement (JPA)
JIRA Process Advancement (JPA)
JIRA OOME (JOOME)
list name
View issue • Add to next sprint
START-85 has been created
MM MM MM
99+122
99+2 12
67%Uploading…
MOVED
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
MOVED
26/11/2015
Due date
November 2015
MON TUE WED THU FRI SATSUN
131
87
1514
2221
29
6
2
9
28
5
1
8
6
13
20
27
5
12
19
26
4
11
18
25
3
10
17
24
2
9
16
23
30
7
3
10
4
11
MOVED
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
NEW
IN PROGRESS
REMOVED
SUCCESS
DEFAULT
MOVED

21. Trust in our skills
Our product decisions are made up of…
Careful thought Research

22. Early signal
testing

23. Agenda
What it is
Early signal methods
Getting started

24. Agenda Early signal methods
Getting started
What it is

25. Agenda Early signal methods
Getting started
What it is

26. Improve itEnvision it Make it

29. Behavioral
Attitudinal
Qualitative (direct)
Ethnographic field studies
Usability lab studies
Quantitative (indirect)
Interviews
Focus groups
A/B testing
Click analysis
Concept testing
Customer feedback
Diary studies
Card sorting
Unmoderated studies
Early signal testing
Early signal testing

30. Gather quick, actionable
feedback from customers
EARLY SIGNAL TESTING

31. No worries
You don’t need to be
a research expert
Why use it?
Guidance
Use information, not
just opinions
Direction
Get cues, not
binding answers
Quick
Don't need detailed
mockups or working
code

32. Methods

33. Aesthetics test
Info test
Expectation test
Preference test
Page design test
Value test
Task completion test
Comprehension test
Navigation test
Usability test
Concept test
Click test
DIFFERENT METHODS

34. Aesthetics test
Info test
Expectation test
Preference test
Page design test
Value test
Task completion test
Comprehension test
Navigation test
Usability test
Concept test
Click test
DIFFERENT METHODS

35. Aesthetics test

36. See if there’s any
unintended reactions
Assess which visual
connects best
Collect first impressions
and emotional responses
Aesthetics test

37. See if there’s any
unintended reactions
Assess which visual
connects best
Collect first impressions
and emotional responses
Aesthetics test

38. See if there’s any
unintended reactions
Assess which visual
connects best
Collect first impressions
and emotional responses
Aesthetics test

42. Preference test

43. Preference test
Build off why users prefer
one option
See if there’s an easy winner
Gather users' initial
inclinations

44. Build off why users prefer
one option
See if there’s an easy winner
Gather users' initial
inclinations
Preference test

45. Build off why users prefer
one option
See if there’s an easy winner
Gather users' initial
inclinations
Preference test

46. Which option do you prefer and why?
A site consists of user groups: admins,
project admins, and users. Organizations
work on projects in a certain area of interest.
Users groups work on and complete these
projects. User groups consist of users.
Admins restrict each user by access or role.
Projects work out specific plans or designs
An organization consists of people.
Organizations work on projects in a certain
area of interest. Teams work on and complete
these projects. Teams consist of members.
Each member fills a role. Projects work out
specific plans or designs.
Option A Option B

48. Which option do you prefer and why?
Option A Option B

50. Comprehension
test

51. Ensure users don’t
misunderstand information
Ensure what you’ve written
is clear
Ensure users comprehend
complicated information
Comprehension
test

52. Ensure users don’t
misunderstand information
Ensure what you’ve written
is clear
Ensure users comprehend
complicated information
Comprehension
test

53. Ensure users don’t
misunderstand information
Ensure what you’ve written
is clear
Ensure users comprehend
complicated information
Comprehension
test

59. Navigation test

60. See if alternate navigation
paths are needed
Assess usability, but don’t
have a full flow yet
Understand users’ natural
wayfinding inclination
Navigation test

61. See if alternate navigation
paths are needed
Assess usability, but don’t
have a full flow yet
Understand users’ natural
wayfinding inclination
Navigation test

62. See if alternate navigation
paths are needed
Assess usability, but don’t
have a full flow yet
Understand users’ natural
wayfinding inclination
Navigation test

66. Ash - we need a screenshot of the icons
in product

67. Concept test

68. Learn what’s valuable,
what’s not, what’s missing
Compare your idea to users’
current process
Explore early product and
feature ideas
Concept test

69. Learn what’s valuable,
what’s not, what’s missing
Compare your idea to users’
current process
Explore early product and
feature ideas
Concept test

70. Learn what’s valuable,
what’s not, what’s missing
Compare your idea to users’
current process
Explore early product and
feature ideas
Concept test

74. Aesthetics test
Info test
Expectation test
Preference test
Page design test
Value test
Task completion test
Comprehension test
Navigation test
Usability test
Concept test
Click test
DIFFERENT METHODS

75. Getting started

76. 5 quick & easy steps
Step 1:
Define your
question &
method
Step 2:
Figure out the
“who” and
the “how”
Step 3:
Create tasks,
questions &
design mocks
Step 4:
Test with
people
Step 5:
Analyze

77. Step 1:
Define your question
& method

78. Does our intent
for our new
logos match
how people will
perceive them?
What do
people think of
the new logos?
NOT THIS THIS

79. Get your team
involved
Speaking with stakeholders before
you run your research will ensure it
hits the mark
#ProTip

80. 5 quick & easy steps
Step 1:
Define your
question &
method
Step 2:
Figure out the
“who” and
the “how”
Step 3:
Create tasks,
questions &
design mocks
Step 4:
Test with
people
Step 5:
Analyze

81. Step 2:
Figure out the “who”
and the “how”

82. Who?

83. How?

84. Unmoderated
Users independently complete tasks,
quick turnaround time
Moderated
Speak with users, gather rich feedback,
ask follow-up questions

85. #ProTip Unmoderated toolkit
Check out Usertesting.com, User
Zoom, Loop11, Validately, and others

87. 5 quick & easy steps
Step 1:
Define your
question &
method
Step 2:
Figure out the
“who” and
the “how”
Step 3:
Create tasks,
questions &
design mocks
Step 4:
Test with
people
Step 5:
Analyze

88. Step 3:
Create tasks, questions
& design mocks

89. Tasks & questions

90. Write scenarios
Rather than just creating tasks, create
one story that ties together all the tasks.
#ProTip

92. Keep it simple
Only design what you need and
don’t overthink it.
#ProTip

93. 5 quick & easy steps
Step 1:
Define your
question &
method
Step 2:
Figure out the
“who” and
the “how”
Step 3:
Create tasks,
questions &
design mocks
Step 4:
Test with
people
Step 5:
Analyze

94. Step 4:
Test with people

95. Don’t just listen
It’s not just what users say,
it’s what they do
#ProTip

96. 5 quick & easy steps
Step 1:
Define your
question &
method
Step 2:
Figure out the
“who” and
the “how”
Step 3:
Create tasks,
questions &
design mocks
Step 4:
Test with
people
Step 5:
Analyze

97. Step 5:
Analyze

98. Task completion
Time to taskUMUX-lite
Time on task
Expectation score
Happy path clicks
20%
Baseline New
100%+80%
74.81 77.90+3.09
1m : 36s 2m : 27s
+51s
5.0 / 7.0 6.3 / 7.0
+3.1
6 3
100%
Baseline New
100%0%
79.74 82.48+2.74
0m : 47s 1m : 00s
+13s
4.6 / 7.0 5.0 / 7.0
+0.4
6 3
Experience
Create a UX scorecard
Metrics like time on task, task completion
or user-reported metrics like ease of use are
valuable comparison points.
#ProTip

99. Task completion
Time to taskUMUX-lite
Time on task
Expectation score
Happy path clicks
20%
Baseline New
100%+80%
74.81 77.90+3.09
1m : 36s 2m : 27s
+51s
5.0 / 7.0 6.3 / 7.0
+3.1
6 3
100%
Baseline New
100%0%
79.74 82.48+2.74
0m : 47s 1m : 00s
+13s
4.6 / 7.0 5.0 / 7.0
+0.4
6 3
- Many decision points for users and a cognitive overload
- General understanding of certain concepts in project
create caused confusion for users i.e. project key
Baseline
- Help and guidance for first time users
- Progressive disclosure of template information
- Template information that means something to users
New experience
What we tested
Experience
Cohort
Existing users
We tested in cohorts of 10 existing JIRA users (admin
and non-admin) per model.
New users
We tested in cohorts of 30 new users per model.
Tasks:
You now want to setup this new project. You need to make sure
that in this new project you can plan, track and report all of your
work. Show how you would setup this new project.
We wanted to know if users can
Easily create a project.
Easily understand where they are.
Easily comprehend and understand the differences between
project templates.
Create a project quickly (no friction points).
UX Scorecard - Creating a project

100. 5 quick & easy steps
Step 1:
Question &
method
Step 2:
Who & how
Step 3:
Mocks &
tasks
Step 4:
Test
Step 5:
Analyze

101. Early signal testing helps inform
your decision but it's not the
decision.
- ASH & BECKY :)

102. The next time you’re facing a
tough decision…

103. Thank you!
ASHLEIGH STERZENBACH | SENIOR DESIGNER | ATLASSIAN
BECKY WHITE | SENIOR RESEARCHER | ATLASSIAN