In this talk, I will describe C-RITE - a user research technique I developed as the head of UX research for Android. My research team at Android not only works on the Android operating system but also Google services (apps, the Play store, Play vertical), Google Now/Voice, Hardware (phones, tablets), Android Wear, Automotive, TV and many other cool projects. Android is a fast-paced environment that ships products on cycles ranging from few weeks to a few months. This method worked well for us because it helped the team to iterate quickly and make big qualitative user experience jumps with confidence.

1. C(ollab) RITE
How to run impactful iterative studies in a fast paced environment
Jhilmil Jain
Android
#crite
@jhilmiljain
#androidux
Wednesday, July 30, 14

2. C-RITE
• Iterative UX research method suitable for agile
environments
• From research to new design solution in 1 day
• 7 research sessions in 2 days
• Collaborative design workshop each day
• Prototype changes between day 1 and 2
• Can be repeated weekly
• Stakeholders collaborate in all UX phases
#crite #androidux
Wednesday, July 30, 14

3. The
Players
The Players and Their Roles
Wednesday, July 30, 14

4. Research Analysis
Reporting
&
Prioritization
Design Prototyping
Planning
& Recruiting
Traditional Roles
Reporting
&
Prioritization
Planning
& Recruiting
Research Analysis Design Prototyping
Reporting
&
Prioritization
Planning
& Recruiting
Designer Designer
Researcher
PM PM
Engineer
Wednesday, July 30, 14

5. Collaboration in C-RITE
Research Analysis Design Prototyping
Reporting
&
Prioritization
Planning
& Recruiting
Designer
PM
Engineer
Researcher
Wednesday, July 30, 14

6. Time Line - Traditional Usability
Research Analysis
Reporting
&
Prioritization
Design Prototyping
Planning
& Recruiting
1- 2 weeks 1- 2 weeks 1/2 - 1week 1/2 -1week
4 - 8 weeks
#crite #androidux
1/2 - 1 week1/2 -1 week
Wednesday, July 30, 14

7. Time Line - RITE
Research Analysis
Reporting
&
Prioritization
Design Prototyping
Planning
& Recruiting
1 - 2 weeks 1 - 2 weeks
2 - 4 weeks
#crite #androidux
Wednesday, July 30, 14

8. Time Line - C-RITE
Research Analysis
Reporting
&
Prioritization
Design Prototyping
Planning
& Recruiting
1-2 weeks 2.5 days
1.5 - 2.5 weeks
#crite #androidux
Wednesday, July 30, 14

9. Time Line - Cyclic C-RITE
Research Analysis
Reporting
&
Prioritization
Design Prototyping
Planning
& Recruiting
1 week 2.5 days
1.5 weeks
#crite #androidux
Wednesday, July 30, 14

10. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
Example: TV Voice Search
Wednesday, July 30, 14

11. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
1st TV Voice Search Design
Wednesday, July 30, 14

12. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Research Timing
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
#crite #androidux
Wednesday, July 30, 14

13. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Collaborative Observation
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
#crite #androidux
Wednesday, July 30, 14

14. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
Observation Board Example
Wednesday, July 30, 14

15. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
User Goals Example
Observation Example
Goal example
P1P1 P2P2
Bob Lin Ali Nur Bob Lin Ali Nur
User knows what to say 2 1 2 2
The system understood user’s intent 5 3 1 1
The system doesn’t slow down user
w unnecessary confirmations 4 4 2 2
User can easily tell where speech
will work
2 2 2 2
User comfortable using their voice 4 4 4 4
Rating on a scale from 1- 5 1 = poor 5 = good
Wednesday, July 30, 14

16. Mocks Example
Wednesday, July 30, 14

17. Mocks Example
Wednesday, July 30, 14

18. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Collaborative Observation
#crite #androidux
• Re-call to count observations, analysis step
reduced
• Disagreements about observations are resolved
immediately
• Research protocol can be adjusted based on
stakeholder feedback
• Researcher can explain what observations
mean
• Small sample size discussion is obsolete
Wednesday, July 30, 14

19. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Analysis
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
2:00 pm 3:00 pm
#crite #androidux
Wednesday, July 30, 14

20. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
From Observation to Insight
Observation Example
Goal example
Wednesday, July 30, 14

21. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Prioritizing Insights
Goal example
Insights
Wednesday, July 30, 14

22. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
3:00 pm
Day 1
10:00 am 11:00 am 1:00 pm 2:00 pm 10:00 am 11:00 am 1:00 pm 2:00 pm
Day 2
6:30 pm5:30 pm
#crite #androidux
Wednesday, July 30, 14

23. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Design Workshop
Wednesday, July 30, 14

24. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Design Workshop Actions
Observation Example
Goal example
Wednesday, July 30, 14

25. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Action Item Owners & Due Dates
Wednesday, July 30, 14

26. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Before & After
Wednesday, July 30, 14

27. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
Wednesday, July 30, 14

28. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
• Designers demonstrate problem solving skills
Wednesday, July 30, 14

29. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
• Designers demonstrate problem solving skills
• Engineers solve technical constraints
Wednesday, July 30, 14

30. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
• Designers demonstrate problem solving skills
• Engineers solve technical constraints
• Researchers guide insight interpretation
Wednesday, July 30, 14

31. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
• Designers demonstrate problem solving skills
• Engineers solve technical constraints
• Researchers guide insight interpretation
• PM gives business and scheduling input
Wednesday, July 30, 14

32. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
• Designers demonstrate problem solving skills
• Engineers solve technical constraints
• Researchers guide insight interpretation
• PM gives business and scheduling input
• Everyone participates in creative process
Wednesday, July 30, 14

33. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Reporting
Day 1
Send action items to team
Day 2
Send action items to team Short report
Day 3
#crite #androidux
Wednesday, July 30, 14

34. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Prototyping
3:00 pm10:00 11:00 1:00 2:00
Day 2
6:30 pm
Day 1
10:00 11:00 1:00 2:00 5:30 pm
#crite #androidux
Wednesday, July 30, 14

35. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Cycles
Week1 Week 2 Week 3
#crite #androidux
Wednesday, July 30, 14

36. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Recruiting
• Create a panel of users that meet
general recruitment criteria
• Panel members commit to availability at
short notice on research days
• Each week, schedule panel members
• Recruit for specific criteria when needed
#crite #androidux
Wednesday, July 30, 14

37. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Preparations
• Set expectations with all stakeholders
beforehand about time commitment,
collaboration and workshop format
• Decision makers need to participate in
C-RITE studies and workshops
#crite #androidux
Wednesday, July 30, 14

38. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
C-RITE Summary
#crite #androidux
• Iterative UX research method suitable for agile
environments
• From research to new design solution in 1 day
• 7 research sessions in 2 days
• Collaborative design workshop each day
• Prototype changes between day 1 and 2
• Can be applied weekly
• Stakeholders collaborate in all UX phases
Wednesday, July 30, 14

39. Thank you
#crite #androidux
Wednesday, July 30, 14

40. Appendix
Wednesday, July 30, 14

41. Agile Methods: RITE
• Changes are made as soon as a problem is
identified, sometimes after 1 session
• 1-2 sessions a day. Breaks between sessions long
enough to make changes
• 1-2 weeks of sessions
• Requires stakeholder attendance and participation
Medlock, Wixon, Fulton, Terrano and Romero, UPA, 2002
#crite #androidux
Wednesday, July 30, 14

42. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
Agile Methods: KRUG + RITE
• 4 participants over 2 days, at least 1 hour
between sessions to make changes
• Researcher sends out observations and
recommendations 30 min after last session
• Team debrief of observations and action items on
day 2, 1 hour after last session
• No collaborative design workshop
McGinn, Chen, Journal of Usability Studies, 2013
Wednesday, July 30, 14

43. Research Analysis
Reporting &
Prioritization
Design Prototyping
Planning
& Recruiting
Agile Methods: PULSE
• 3 participants, 1 day every week
• Analysis and write-up of results by researcher
within 2 days
• No collaborative analysis and design workshop
Konno, Agile 2012
Wednesday, July 30, 14

44. Research Analysis
Reporting &
Prioritization
Design Implementation
Planning
& Recruiting
Variations We’ve Tried
• C-RITE Remote
• If time zones allow, stakeholders participate through ‘Hangouts’
• Sessions are recorded and watched with time delay
• Next day debriefs and design workshops for incompatible time
zones
• C-RITE Diary Studies
• In-lab diary kick-offs and debriefs are run in C-RITE format
• Researcher analyzes and summarizes diary data for debrief and
design workshop
Wednesday, July 30, 14