Professor Jeanne Liedtka and Associate Professor Kristina Jaskyte Bahr unpack the results and learnings from design thinking impact assessments and offer the tool again so you can participate if you missed out the first time.

2. INDUSTRIES TOP 3 ROLES TOP 3 SENIORITYEMPLOYEE RANGES
WEBINAR AUDIENCE BREAKDOWN

3. Why Design Thinking?
REPORTING ON RESULTS FROM A
NEW MEASUREMENT TOOL

4. RESEARCH QUESTIONS
1. What are the actual organizational practices in use that constitute “design
thinking”
2. What kinds of outcomes are DT practitioners observing, related to its use?
3. What if any, are the significant relationships between particular DT
practices and DT outcomes?
4. Does the organization type and level of experience with Design Thinking
practice influence the relationship between DT practices and DT
outcomes?

5. MEASURING DT’S IMPACT IS COMPLICATED

6. 1. Allows us to tie DT to behaviors not just financial flows
2. Encourages us to think about the metrics we need, rather than
the ones we’ve got
3. Shows us what deep DT done well should look like in action
4. Gives us a vehicle for measuring direct before & after on
things we care about
5. Connects us to decades of existing research in the social sciences
OUR APPROACH: DECONSTRUCT ELEMENTS OF DT
WHY?

7. DECONSTRUCT THE OUTCOMES ELEMENTS PRODUCE

8. SURVEY: SECTION 1. DESIGN THINKING PRACTICES
11 ITEMS

9. SURVEY: SECTION 2. DESIGN THINKING OUTCOMES
41 ITEMS

10. • 3 Waves of data gathering:
- 104 Responses / 75 Surveys completed
(DTDC meet-up group/ Stanford D-School trainers)
- 325 Responses / 231 Completed
(corporate clients of MURAL virtual software)
- 221 Responses / 165 Completed
(graduates of Darden DT Online courses)
• Organization type:
Business (N=250), Nonprofit (N=68), Government (N=59)
• Level of participants’ expertise in using Design Thinking process:
None (1.1%), Some (15%), Moderate (50.4%), Extensive (33.5%)
SAMPLE DETAILS

11. RESEARCH QUESTIONS
1. What are the actual organizational practices in use that constitute “design
thinking”

12. DISCOVERY AND IDEATION
Followed a structured process
Done user research using ethnographic
tools (e.g., interviewing and
observation, journey mapping, job-to-
be-done, etc.)
Focused your problem definition on the
user's perspective rather than the
organization's
Created a set of design criteria that
described an ideal solution, based on
user research
Generated a diverse set of ideas based
on your user research
TEAM FORMATION AND
FUNCTIONING
Formed a diverse team
Emphasized active listening
among team members in order
to find shared meaning
PROTOTYPING AND
EXPERIMENTATION
Created prototypes of your
ideas (e.g., storyboards, videos,
mock-ups of offerings)
Moved multiple ideas into
prototyping and testing
Got feedback from users and
other stakeholders on the
prototype
Executed real world
experiments to test your ideas
DESIGN THINKING PRACTICES
EXPLORATORY FACTOR ANALYSIS RESULTS

17. RESEARCH QUESTIONS
1. What are the actual organizational practices in use that constitute “design
thinking”
2. What kinds of outcomes are DT practitioners observing, related to its use?

18. DT
outcomes
Improved
Implementati
on and
Adaptation
Individual
Psychological
Benefits
Increased
Solution
Quality
Network
Capability
and Resource
Enhancement
Trust building
DESIGN THINKING OUTCOMES
EXPLORATORY FACTOR ANALYSIS RESULTS

19. Factor 1: Improved Implementation and Adaptation Factor 2: Individual
Psychological Benefits
Factor 3: Network Capability and
Resource Enhancement
• Enhanced your ability to pivot when initial solution didn't
work
• Improved the likelihood of the implementation of new
solutions
• Made it easier to discard solutions that didn't work as
planned
• Encouraged shifts in organizational cultural that made it
more customer-focused
• Encouraged changes in organizational culture that
made risk-taking more acceptable
• Kept people motivated to work on a project to achieve
impact
• Broadened organization's definition of what innovation is
• Increase a sense of ownership and acceptance of a
solution
• Increased appreciation for use of data to help drive
decisions
• Created a sense of safety to try
new things
• Gave employees more confidence
in their own creative abilities
• Helped people interested in trying
new things to connect and support
each other
• Encouraged people's open-
mindedness to try new things
• Built new relationships locally that continued
after the initial project was completed
• Expanded access to new resources for
individuals and teams
• Helped pool resources for greater impact
• Enhanced other stakeholders' willingness to
collaborate on new solutions
Factor 4: Increased Solution Quality Factor 5: Trust Building
• Helped teams see the problems in new ways, resulting
in solving more promising problem
• Increased engagement of employees involved in the
Design thinking process
• Allowed new and better solutions, not visible at the
beginning of the process, to emerge during it
• Fostered the inclusion of user input
• Helped people involved to examine their own biases
and preconceptions
• Built trust among team members
• Built trust between problem-solving
teams and other stakeholders

22. RESEARCH QUESTIONS
1. What are the actual organizational practices in use that constitute “design
thinking”
2. What kinds of outcomes are DT practitioners observing, related to its use?
3. What if any, are the significant relationships between particular DT
practices and DT outcomes?

23. Research Question 3:
What are the relationships between the tools/activities and the outcomes? Are
particular outcomes associated with particular tools/activities?
DT
Practices
Improved
Implementation and
Adaptation
Individual
psychological
benefits
Network Capability
and Resource
Enhancement
Increased
Solution Quality
Trust building
Followed a structured process
Formed a diverse team X X X X X
Emphasized active listening X X X X X
Done user research X negative
Focused problem definition on
user’s perspective
X X X
Created a set of design criteria X X
Generated a diverse set of
ideas
X X X
Created prototypes
Moved multiple ideas into
testing
Got feedback from users X
Executed real world
experiments
X X X X

24. RESEARCH QUESTIONS
1. What are the actual organizational practices in use that constitute “design
thinking”
2. What kinds of outcomes are DT practitioners observing, related to its use?
3. What if any, are the significant relationships between particular DT
practices and DT outcomes?
4. Does the organization type and level of experience with Design Thinking
practice influence the relationship between DT practices and DT
outcomes?

25. • Formed a diverse team (a significant predictor of all 5 Design Thinking
outcomes)
• Emphasized active listening (a significant predictor of all 5 Design Thinking
outcomes)
• Executed real world experiments (a significant predictor of 4 out of 5
Design Thinking outcomes)
• Focused problem definition on user’s perspective (a significant predictor of
3 out of 5 practices)
• Generated a diverse set of ideas (a significant predictor of 3 out of 5
practices)
FIVE “SUPER PRACTICES” IDENTIFIED

27. Expertise-related Movements with significance
DT outcome from no
experience to
limited experience
from limited to
moderate level
from moderate
level to extensive
experience
Improved implementation and
adaptation
Individual psychological
benefits
Network capability and resource
enhancement
Increased solution quality
Trust building
DATA SUGGESTS THERE ARE THRESHOLDS
FOR THE VALUE OF INVESTING IN INCREASING EXPERTISE

28. AND SO…PRELIMINARY OBSERVATIONS
• There exists a widely shared set of practices that operate under the umbrella of “Design
Thinking”. The use of those practices, in general, did not vary significantly across business,
nonprofit, and government organizations.
• An exploratory factor analysis produced a three factor Design Thinking Practices solution, with
three focusing on Discovery and Ideation, Team building and functioning, and Prototyping and
experimentation.
• An exploratory factor analysis produced a five factor Design Thinking Outcomes solution, with
five factors focusing on Increased solution quality, Individual Psychological benefits, Improved
implementation and adaptation, Network capability and resource enhancement, and Trust
building.
• The use of specific 11 Design Thinking practices correlated with each of the five Design thinking
outcomes. Five “super” practices - Formed a diverse team, Emphasized active listening, Executed
real world experiments, Focused problem definition on user’s perspective, and Generated a
diverse set of ideas - were among the most frequent significant predictors of DT outcomes.
• The value of experience appears to have thresholds for most factors.

29. Q&A

30. 1. Begin to systematically think thru the possible universe of outcomes
2. Identify where you are currently performing well to tell your story
3. Identify where you are missing opportunities for impact
4. Utilize pre-test/post-test measures to demonstrate impact
HOW CAN YOU USE THE NEW MEASUREMENT
INSTRUMENT?