Slides as presented at the XP2020 Conference (Copenhagen/Online) by Matthew Philip (Accenture | SolutionsIQ). Abstract Hypothesis-Driven Development is thinking about the development of new ideas, products and services – even organizational change – as a series of experiments to determine whether an expected outcome will be achieved, so we need to know how to design and run experiments properly. This workshop session helps participants understand the importance of using experiments to help teams and organizations learn and improve, while giving hands-on practice in designing experiments to yield measurable evidence for that learning. We’ll even play a couple of inductive-logic games to get us in the mindset of how to write tests to validate — and invalidate — our hypotheses in order to acquire knowledge. Whether you’re in product development of organizational improvement, for those wanting to move from projects to experiments in the quest of continuous learning and evolutionary improvement, this session will provide the tools and mindset for going beyond simply calling something an experiment to conducting experiments with the proper rigor to maximize learning. This 60-minute workshop session helps participants: understand the importance of using experiments to help teams and organizations learn and improve gain hands-on practice in designing experiments to yield measurable evidence for that learning understand how to properly measure outcomes without confirmation bias https://www.agilealliance.org/xp2020/xp-2020-online-program/industry-and-practice-abstracts/#Philip

1. @mattphilip #xp2020 1
NO (LAB)
JACKET
REQUIRED
DESIGNING EXPERIMENTS
FOR LEARNING

2. @mattphilip #xp2020 2

3. @mattphilip #xp2020 3
“LET’S EXPERIMENT!”

4. @mattphilip #xp2020 4
WHY EXPERIMENTS?
In complex environments, you can’t follow recipes or
conduct detailed analysis to understand the situation.
Rather, you must experiment (probe).
Cynefin
Improve collaboratively,
evolve experimentally. Kanban Method
Pivoting …. a structured course correction designed to test …
hypothesis about the product, strategy, and engine of growth.
Lean Startup
Each feature includes a benefit hypothesis. SAFe

5. @mattphilip #xp2020 5
WHY EXPERIMENTS?
• Allow ideas that are
not useful to fail in
small, contained
and tolerable ways
(Dave Snowden)
• Make decisions in
situations of high
uncertainty.

6. @mattphilip #xp2020
ELEUSIS EXPEDITIOUS:
A GAME ABOUT HYPOTHESES
6

7. @mattphilip #xp2020
DEBRIEF:
WHAT DID WE LEARN?
7

8. @mattphilip #xp2020
LEAN MASTERMIND
(YOU CAN PLAY IT LATER :) )
8

9. @mattphilip #xp2020 9
SCIENTIFIC METHOD
Define a question “What is the rule?"
Gather information and resources
(observe)
“This card apparently follows the
rule, this other one doesn’t.”
Form an explanatory hypothesis “The rule is only number cards.”
Test hypothesis by experimenting
(replicating)
“Numbers 1-10 would follow rule,
face cards would not.”
Analyze the data “2 and 6 followed the rule, but 5 did
not.”
Interpret and form new hypothesis “The rule is only even numbers.”
Share results and learnings

10. @mattphilip #xp2020
“WELL I AIN'T OFTEN RIGHT,
BUT I'VE NEVER BEEN WRONG.”
— GRATEFUL DEAD
10

11. @mattphilip #xp2020 11
INFORMATION THEORY
Probability of Success
Information
0% 100%50%

12. @mattphilip #xp2020 12

13. @mattphilip #xp2020 13

14. @mattphilip #xp2020
'THE ONLY REAL RISK IS
THINKING YOU HAVE A
WINNING STRATEGY WHEN YOU
HAVE A LOSING ONE.”
— DAN VACANTI
14

15. @mattphilip #xp2020 15
WHAT IS “DONE” FOR YOU?
-- John Cutler

16. @mattphilip #xp2020 16

17. @mattphilip #xp2020 17
HYPOTHESIS-DRIVEN DEVELOPMENT

18. @mattphilip #xp2020 18
EXPERIMENT A3
Karl Scotland Ltd Experiment A3 https://availagility.co.uk/templates
Context Our problem is… Actions
Hypothesis We believe we can solve it by… Results
Failure
Rationale We believe this because… Follow-up
Failure
Title: Owner:
Success
Success
To prove/disprove the hypothesis we will…
We will declare success or failure when…
As a result of success or failure we will…

19. @mattphilip #xp2020 19
GENERATING EXPERIMENTS

20. @mattphilip #xp2020
LET’S PRACTICE!
20

21. @mattphilip #xp2020 21
• Company policy to give a bonus to employees who refer new hires
• Implementing SAFe
• Rolling out a new team-leadership model
• Work-from home policy
• [Your own example]
HOW WOULD YOU TEST THESE?

22. @mattphilip #xp2020 22
• Time-based
• Measurable
• Single variable (Control!)
• Provable (and disprovable) hypothesis
• Psychological safety to fail
• Learning is the outcome
ELEMENTS OF GOOD EXPERIMENTS

23. @mattphilip #xp2020 23
TYPES OF FAILURE
Preventable Complex Intelligent
Definition Deviations from known
process that produce
unwanted outcomes
Unique and novel
combinations of events
and actions that give
rise to unwanted
outcomes
Novel forays into new
territory that lead to
unwanted outcomes
Common Causes Behavior, skill, attention
deficiencies
Complexity, variability,
novel factors imposed
on familiar situations
Uncertainty,
experimentation, risk
taking
AKA Process deviation System breakdown Unsuccessful trial
Contexts Where Each
Is Most Salient
Production line
manufacturing
Fast-food services
Basic utilities and
services
NASA shuttle program
Aircraft carrier
Nuclear power plant
Drug development
New product design
From Amy Edmondson, The Fearless Organization: Creating Psychological Safety in the Workplace for Learning, Innovation, and Growth

24. @mattphilip #xp2020 24
PRODUCTIVE RESPONSES TO FAILURE
Preventable Failure Complex Failure Intelligent Failure
• Training
• Retraining
• Process improvement
• System redesign
• Sanctions, if repeated or
otherwise blameworthy
actions are found
• Failure analysis from diverse
perspectives
• Identification of risk factors to
address
• System improvement
• Failure parties
• Failure awards
• Thoughtful analysis of results
to figure out implications
• Brainstorming of new
hypotheses
• Design of next steps or
additional experiments
From Amy Edmondson, The Fearless Organization: Creating Psychological Safety in the Workplace for Learning, Innovation, and Growth

25. @mattphilip #xp2020 25
DESTIGMATIZING FAILURE
Traditional Frame Destigmatizing Frame
Concept of Failure Failure is not acceptable. Failure is a natural by-product of
experimentation.
Beliefs About Effective
Performance
Effective performers don’t fail. Effective performers produce, learn
from and share the lessons from
intelligent failures.
Goal Prevent failure. Learn fast.
Impact of Frame People hide failures to protect
themselves.
Open discussion, fast learning and
innovation.
From Amy Edmondson, The Fearless Organization: Creating Psychological Safety in the Workplace for Learning, Innovation, and Growth

26. @mattphilip #xp2020 26
• Language: “Fail Fast” ! “Prove/disprove hypothesis”
• Leaders SETT the boundaries (“Safe Enough To Try”)
• Organizational experiments board
• “Goldratt User Stories”
• Finding your optimal WIP limit
• Retrospective actions
• A/B tests
OTHER CONSIDERATIONS

27. @mattphilip #xp2020
“WE DO NOT DO PROJECTS ANYMORE,
ONLY EXPERIMENTS.”
— BARRY O’REILLY
27

28. @mattphilip #xp2020 28
“CURATOR OF EXPERIMENTS”

29. @mattphilip #xp2020 29
• Eleusis Expeditious and Eleusis Express (https://availagility.co.uk/resources/games/eleusis-expeditious/)
• A3 Experiment Canvas from Karl Scotland (https://availagility.co.uk/)
• Safe-to-Fail Probes (http://cognitive-edge.com/methods/safe-to-fail-probes/)
• How to Fail like a Pro, Freakonomics Radio: 370.
• Unlearn by Barry O’Reilly
• Company-wide Agility with Beyond Budgeting, Open Space & Sociocracy by John Buck and Jutta Eckstein
• Principles of Product Development Flow by Don Reinertsen
• How to Measure Anything by Douglas W. Hubbard
• pretty much anything from John Cutler
• The Surprising Power of Online Experiments (https://hbr.org/2017/09/the-surprising-power-of-online-experiments)
• Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 years (http://bit.ly/KDD2015Kohavi)
• Thinking in Bets by Annie Duke
• How to Implement Hypothesis-Driven Development (https://barryoreilly.com/how-to-implement-hypothesis-driven-development/)
• Beyond the Black Hole: Product Management for Continuous Delivery, Elizabeth Ayer, FlowCon 2019
• The Fearless Organization by Amy Edmondson
• Don’t Be a Ditka, Dan Vacanti (https://www.infoq.com/presentations/prioritization-scarcity-stress-uncertainty/)
• Goldratt User Stories (https://yorkesoftware.com/2018/03/09/goldratt-user-stories/)
• Robert Hooke, first Curator of Experiments (https://royalsociety.org/science-events-and-lectures/2003/summer-science/mr-hooke/)
SOURCES AND RESOURCES
MATTPHILIP.WORDPRESS.COM | MATTHEW.R.PHILIP@ACCENTURE.COM

30. @mattphilip #xp2020 30
THANK
YOU
www.solutionsiq.com