Slides as presented during the workshop with Karl Scotland at Flowcon, Paris, Dec. 12, 2019. https://www.flowcon.fr/
Hypothesis-Driven Development is thinking about the development of new ideas, products and services – even organizational change – as a series of experiments to determine whether an expected outcome will be achieved, so we need to know how to design and run experiments properly.
This session helps participants understand the importance of using experiments to help teams and organizations learn and improve, while giving hands-on practice in designing experiments to yield measurable evidence for that learning. We’ll even play a couple of inductive-logic games to get us in the mindset of how to write tests to validate – and invalidate – our hypotheses in order to acquire knowledge.
Whether you’re in product development or organizational improvement, for those wanting to move from projects to experiments in the quest of continuous learning and evolutionary improvement, this session will provide the tools and mindset for going beyond simply calling something an experiment, to conducting experiments with the proper rigor to maximize learning.
5. @kjscotland #FlowCon @mattphilip
WHY EXPERIMENTS?
In complex environments, you can’t follow recipes
or conduct detailed analysis to understand the
situation. Rather, you must experiment (probe).
Cynefin
Improve collaboratively,
evolve experimentally. Kanban Method
Pivoting …. a structured course correction
designed to test … hypothesis about the
product, strategy, and engine of growth.
Lean Startup
Each feature includes a benefit hypothesis. SAFe
Complex Complicated
Simple
Chaotic
Relationship between
cause and effect clear
only in hindsight
Relationship between
cause and effect
requires analysis
Relationship between
cause and effect
obvious to all
No relationship between
cause and effect at
system level
“If … maybe?” “If… then… then”
“Now that…” “If… then”
7. @kjscotland #FlowCon @mattphilip
IDEA
◆ This is a variation on Eleusis Express; a card game of inductive reasoning which itself is a
modification of Robert Abbott’s game Eleusis.
◆ One player has a secret rule for which cards can be played.
◆ For example: Each card has to be a different color from the card before it.
◆ Other players lay down cards to discover the pattern. If a player lays down a card correctly,
he or she can try to guess the rule.
8. @kjscotland #FlowCon @mattphilip
OBJECT
◆ A game consists of one or more rounds.
◆ A different player is chosen as oracle of each new round, and it is the oracle who knows the
secret rule. The oracle does not play any cards that round.
◆ All plays are made to a central layout that grows as the round progresses. A layout consists
of a horizontal mainline of correct cards that follow the secret rule. Below the mainline are
vertical sidelines of cards that did not follow the rule. These sideline cards can be
overlapped to save space.
9. @kjscotland #FlowCon @mattphilip
ORACLE
◆ The oracle has a secret rule.
◆ He or she places the pile of cards face down, turns over the top card, and puts it on the
table. This will be the start of the mainline.
◆ Before play starts, the oracle may give a hint about the rule.
◆ The player to the left of the oracle goes first, then the play continues around to the left.
10. @kjscotland #FlowCon @mattphilip
PLAYERS
◆ A player turns the top card from the deck and lays it on the table. He or she asserts whether
it is correct or not, and the oracle accepts or rejects.
◆ If correct, the card goes to the right of the last card on the mainline.
◆ If incorrect, the card goes below the last correct card (it either starts a sideline or it adds to
a sideline).
11. @kjscotland #FlowCon @mattphilip
GUESSING
◆ Whenever a player makes a correct assertion, he or she can guess the rule.
Everyone must hear the guess.
◆ The oracle then says whether the player is right or wrong. If wrong, the game
continues. If right, the round ends.
14. @kjscotland #FlowCon @mattphilip
1. Choose who is who. One player becomes the codemaker, the others are the
codebreakers
2. Choose 4 digits. The codemaker chooses a pattern of four digits; for example 1 2 3 4.
3. Try guessing the pattern. The codebreakers try to guess the pattern, in both order
and number, within 12 turns. Each guess is made by writing a row of four.
4. Provide feedback. After each guess the codemaker provides feedback by placing from
zero to four key dots next to the guess in the following order.
1. A full key dot [●] indicates the existence of a code number which is correct in both number and position.
2. An empty dot [○] indicates the existence of a correct number placed in the wrong position.
5. Keep guessing. Once feedback is provided, another guess is made. Guesses and
feedback continue until either the codebreakers guess correctly, or 12 incorrect
guesses are made.
INSTRUCTIONS
15. @kjscotland #FlowCon @mattphilip
As before, with the following additions:
• Before a guess: Write down the intent of the guess –
what it is you want to learn.
• After feedback: Write down the result and what you
actually learned.
WORKSHEET
16. @kjscotland #FlowCon @mattphilip
RULES (FOR ORACLE’S EYES ONLY)
1.If the last card was red, play a black card. If the last card was black, play a red card.
2.If the last card was a spade, play a heart; if last card was a heart, play a diamond; if last was
diamond, play club; and if last was club, play a spade.
3.The cards on the mainline must follow this pattern: three red cards, then three black, then three
red, then three black, etc.
4.If the last card is an odd-numbered card, play an even-numbered card; if the last is even, play an
odd. [When numbers are involved, ace is 1 (odd), jack is 11 (odd), queen is 12 (even), and king is 13
(odd).
5.If the last card is among the cards ace to 7, play a card 8 to king. If last is among 8 to king, play ace
to 7.
6.Play a card with a number that is 1, 2, or 3 higher than the number of the last card. The numbers
can “turn-the-corner.”
7.If the last card is an odd-numbered card, play a red card. If the last card is even, play a black card.
8.The card played must be the same suit or the same number as the last card.
9.If the last card is black, play a card with a number that is equal to or lower than the number of the
last card. If the last card is red, play a card equal to or higher than the last card.
18. @kjscotland #FlowCon @mattphilip
• Define a question
• Gather information and resources (observe)
• Form an explanatory hypothesis
• Test the hypothesis by performing and experiment
and collecting data in a replicable manner
• Analyze the data
• Interpret the data and draw conclusions that serve as
a starting point for a new hypothesis
• Publish results
SCIENTIFIC METHOD
30. @kjscotland #FlowCon @mattphilip
• Company policy to give a bonus to employees who
refer new hires
• Implementing SAFe
• Rolling out a new team-leadership model
• Work-from home policy
• [Your own example]
HOW WOULD YOU TEST THESE?
31. @kjscotland #FlowCon @mattphilip
• Psychological safety to fail
• Learning is the outcome
• Time-based
• Measurable
• Single variable (Control!)
• Provable (and disprovable) hypothesis
ELEMENTS OF GOOD EXPERIMENTS
36. @kjscotland #FlowCon @mattphilip
SOURCES AND RESOURCES
◆ Karl Scotland (https://availagility.co.uk/)
◆ https://yorkesoftware.com/2018/03/09/goldratt-user-stories/
◆ http://cognitive-edge.com/methods/safe-to-fail-probes/
◆ Eleusis Express http://www.logicmazes.com/games/eleusis/express.html
◆ Freakonomics Radio: 370. How to Fail like a Pro
◆ Unlearn book by Barry O’Reilly
◆ Agile BOSSA nova book
◆ How to Measure Anything book
◆ Pretty much anything from John Cutler
◆ The Surprising Power of Online Experiments (https://hbr.org/2017/09/the-surprising-power-of-online-
experiments)
◆ https://exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf
◆ http://www.theagilefactor.com/
◆ Thinking in Bets by Annie Duke
◆ https://barryoreilly.com/how-to-implement-hypothesis-driven-development/
◆ Online Experimentation at Microsoft
◆ Elizabeth Ayer, FlowCon 2019 - Beyond the Black Hole: Product Management for Continuous Delivery
◆ http://mattphilip.wordpress.com