Presented at EuroIA17, September 2017; World IA Day NYC, February 2017; Interact, October 2016 (London, UK); earlier versions in 2014 at UXPA Boston (Boston, MA, USA); in 2013 at Interaction S.A. (Recife, Brasil), Intuit (Mountain View, CA, USA), Designers + Geeks (New York, USA); in 2012 at UX Russia (Moscow, Russia), UX Hong Kong (Hong Kong, China), WebVisions NYC (New York, NY, USA); in 2011 at the IA Summit (Denver, CO, USA), UX-LX (Lisbon, Portugal), Love at First Website (Portland, OR, USA). This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

1. Beyond User Research
Lou Rosenfeld • lou@rosenfeldmedia.com
#EuroIA17 • 30 September 2017

2. (a talk has no name)
Lou Rosenfeld • lou@rosenfeldmedia.com
EuroIA 2017 • 30 September 2017

4. What might
we become?

7. What does victory look like?

8. User research in
today’s organization

9. Reports from the
user research group

10. Query data from the
search team
XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800]
"GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%
3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&clie
nt=www&oe=UTF-8&proxystylesheet=www&q=lincens
e+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0
0.02
XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800]
"GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%
3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+pl

11. Logs from the
call center

12. Reports from
analytics applications

13. “Learnings” from
Voice of the Customer research

14. Reports from CRM applications

15. Surveys behind
Net Promoter Score

16. Studies from the
research center

17. Analysis of
social media

18. Research on
brand architecture

19. Even users’
mental models

21. Yet why does
so much design
still SUCK?

22. Yet why does
so much design
still SUCK?

23. Problem
It’s only going
to get worse

24. The blind men and the
elephant

25. Problem
It’s only going
to get worse
Opportunity
How might we support
synthesis and insight?

26. What Why

27. Methods employed:
quantitative versus qualitative

28. Goals: help org or users
Organizational goals
Users’ goals

29. How they use data: measur
world we know versus wo
we don’t
Measuring the world we know
Exploring the world we don’t

30. ...I
Descriptive data
Statistical data

31. This is true.
And so is this.

32. This is true.
And so is this.

33. Lou’s TABLE OF
OVERGENERALIZED
DICHOTOMIES
Web Analytics User Experience
What they
analyze
Users' behaviors (what's
happening)
Users' intentions and
motives (why those things
happen)
What methods
they employ
Quantitative methods to
determine what's happening
Qualitative methods for
explaining why things
happen
What they're
trying to achieve
Helps the organization meet
goals (expressed as KPI)
Helps users achieve goals
(expressed as tasks or
topics of interest)
How they use
data
Measure performance (goal-
driven analysis)
Uncover patterns and
surprises (emergent
analysis)
What kind of data
they use
Statistical data ("real" data
in large volumes, full of
errors)
Descriptive data (in small
volumes, generated in lab
environment, full of errors)

34. Five themes for getting to
synthesis and insight
1. Balance
2. Cadence
3. Conversation
4. Perspective
5. Operations

35. 1. Balance

36. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /
search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&
ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyle
sheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /
search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&
ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i
p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
Web analytics asks:
“Are we converting license plate renewals?”
User researcher asks:
“What are people searching the most?”
Balanced analysis

37. Thanks
Balance within methods
example courtesy to Angel Brown, Ogilvy DigitalHealth

38. Thanks
Balance within methods
example courtesy to Angel Brown, Ogilvy DigitalHealth

39. Rohrer’s user research
landscape
Text
Christian Rohrer: http://www.nngroup.com/articles/which-ux-research-methods/
Balance within practice

40. Rohrer’s user research
landscape
Text

41. 2. Cadence

42. A research cadence
from Whitney Quesenbery

43. Cadence
Weekly
Call center data trend analysis 2 – 4 hours behavioral + quantitative
Task analysis 4 – 6 hours behavioral + quantitative
Quarterly
Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative
User survey 16 – 24 hours attitudinal + quantitative
Annually
Net Promoter Score study 3 – 4 days attitudinal + quantitative
Field study 4 – 5 days behavioral + qualitative

44. Cadence
Weekly
Call center data trend analysis 2 – 4 hours behavioral + quantitative
Task analysis 4 – 6 hours behavioral + quantitative
Quarterly
Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative
User survey 16 – 24 hours attitudinal + quantitative
Annually
Net Promoter Score study 3 – 4 days attitudinal + quantitative
Field study 4 – 5 days behavioral + qualitative
Cadence + Balance

45. Cadence
Weekly
Call center data trend analysis 2 – 4 hours behavioral + quantitative
Task analysis 4 – 6 hours behavioral + quantitative
Quarterly
Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative
User survey 16 – 24 hours attitudinal + quantitative
Annually
Net Promoter Score study 3 – 4 days attitudinal + quantitative
Field study 4 – 5 days behavioral + qualitative
Cadence + Balance

46. 3. Conversation

47. Buy Candy
for Strangers

48. Tell
Stories

49. Tell
Stories
SKU: #39072-2AH1

50. Ban words that
impede conversations
• Product names: Omniture,, SharePoint...
• Methods: focus group,, usability test...
• Departments: market research,, analytics...
• Disciplines: business analysis,, information
architecture...
• Outcomes: portal, social media layer...

51. Consider
framing

52. Consider
framing
Big Data

53. Consider
framing
Big Data THICk Data

54. Consider
framing
Big Data THICk Data
SeeTriciaWang’s
“Why Big Data
NeedsThick Data”:
http://bit.ly/23E9qlv

55. Develop a
pidgin
Dave Gray’s boundary matrix: http://bit.ly/gWoZQm
KPI
goals
segments
personas

56. 4. Perspective

57. Maps help us make sense by
seeing things in new ways

58. Rohrer’s user research
landscape
Text

59. Kaushik’s Trinity Strategy

60. Kaushik’s Trinity Strategy

61. Kaushik’s Trinity Strategy
Avinash Kaushik’s “Trinity:A Mindset & Strategic Approach“: http://bit.ly/2yKA9CC
“...while I have a bucket for ‘Voice of
Customer,’ in hindsight I should have
worked harder still to paint the full qual
and quant picture….”
—Kaushik (in private email)

62. Containers help us make sense
by putting things together
in new ways

63. MailChimp’s UX team:
drowning in data

64. MailChimp + Evernote
• Shared bucket of buckets (60 notebooks)
• Email is the API
• OCR’d (nice for SurveyMonkey reports)
• Searchable!
• Led to “regular data nerd lunches”
MailChimp: on the threshold of
synthesis

65. MailChimp + Evernote
• Shared bucket of buckets (60 notebooks)
• Email is the API
• OCR’d (nice for SurveyMonkey reports)
• Searchable!
• Led to “regular data nerd lunches”
MailChimp: on the threshold of
synthesis was getting closer to

66. 5. Operations

67. WeWork’s Polaris:
working with a blank slate

68. Research challenges
at WeWork
1. Siloed research
2. Gaps in research memory
3. Reports instead of insight

69. WeWork: "nuggetization" +
metadata

70. WeWork: "nuggetization" +
metadata
nuggets
metadata

71. WeWork: "nuggetization" +
metadata

72. WeWork: "nuggetization" +
metadata

73. WeWork: filter/search this stuff
•

74. WeWork: an insight(nuggets)

75. WeWork: "nuggetization" +
metadata
nuggets
metadata

76. WeWork’s approach
• Atomic units smaller than reports—nuggets
(350 interviews yield @3400 nuggets)
• LOTS of metadata
• Findability improves organizational research
memory
• Researcher:curator ratio is 3:1

77. WeWork’s approach
• Atomic units smaller than reports—nuggets
(350 interviews yield @3400 nuggets)
• LOTS of metadata
• Findability improves organizational research
memory
• Researcher:curator ratio is 3:1
…is an IA approach…

78. WeWork’s approach
• Atomic units smaller than reports—nuggets
(350 interviews yield @3400 nuggets)
• LOTS of metadata
• Findability improves organizational research
memory
• Researcher:curator ratio is 3:1
…is an IA approach…
…and an Ops approach

79. Early efforts to operationalize insight

80. Early efforts to operationalize insight

81. Early efforts to operationalize insight

82. Early efforts to operationalize insight

83. Early efforts to operationalize insight

84. Early efforts to operationalize insight
information architects in Wellington interaction designers in London
industrial designers in Seattle
market researchers in London

85. from DevOps: https://devops.com/2014/04/07/evolve-devops/
DevOps => DecisionOps
DevOps

86. from DevOps: https://devops.com/2014/04/07/evolve-devops/
DevOps => DecisionOps
DevOps DesignOps

87. from DevOps: https://devops.com/2014/04/07/evolve-devops/
DevOps => DecisionOps
DevOps DesignOps ResearchOps

88. from DevOps: https://devops.com/2014/04/07/evolve-devops/
DevOps => DecisionOps
DevOps DesignOps ResearchOps InsightOps

89. from DevOps: https://devops.com/2014/04/07/evolve-devops/
DevOps => DecisionOps
DevOps DesignOps ResearchOps InsightOps
Operations is nascent
CreativeOps, Social Media Ops, DataOps…

90. Operations is…

91. Operations is…
a platform
(systems • infrastructure • processes • tooling • principles)

92. Operations is…
a platform
(systems • infrastructure • processes • tooling • principles)
that enables and amplifies the “talent”

93. Operations is…
a platform
(systems • infrastructure • processes • tooling • principles)
that enables and amplifies the “talent”
maximizes efficiency

94. Operations is…
a platform
(systems • infrastructure • processes • tooling • principles)
that enables and amplifies the “talent”
maximizes efficiency
and makes sense of the unknown

95. Operations is…
a platform
(systems • infrastructure • processes • tooling • principles)
that enables and amplifies the “talent”
maximizes efficiency
and makes sense of the unknown

96. Operations requires IA
to enable

97. Operations requires IA
to enable
balance

98. Operations requires IA
to enable
balance
cadence

99. Operations requires IA
to enable
balance
cadence
conversation

100. Operations requires IA
to enable
balance
cadence
conversation
perspective

101. Operations requires IA
to enable
balance
cadence
conversation
perspective

103. What might
we become?

106. Is Information Architecture
more Operations
than anything else?

107. Architecting for insight
Lou Rosenfeld • lou@rosenfeldmedia.com
#EuroIA17 • 30 September 2017

108. Thanks!
slides:
http://rfld.me/11FrI3o
related article:
http://rfld.me/145ZccP
Lou Rosenfeld
@louisrosenfeld
www.rosenfeldmedia.com • @rosenfeldmedia