An introduction to Specification by Example.

1. Specification By
Example
Declan Whelan
@dwhelan

2. What’s the Problem?

3. Waterfall
Source: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

4. Waterfall
X
Source: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

5. Cost Of Change
Cost%
Of%
Change%
Time%

6. Agile Specification
Agile&Approach&
Itera/on&1& Itera/on&2& Itera/on&3& Itera/on&4&
Requirements& Requirements& Requirements& Requirements&
Analysis&&&Design& Analysis&&&Design& Analysis&&&Design& Analysis&&&Design&
Cost%
of% Code& Code& Code& Code&
Change%
Test& Test& Test& Test&
Deploy&
Time%

7. Traditional Flow

8. Traditional Flow
Specification

9. Traditional Flow
Build
Specification

10. Traditional Flow
Build
Specification
Tests Cases

11. Traditional Flow
Build
Specification Test
Tests Cases

12. Traditional Flow
Build
Specification Test
Deploy
Tests Cases

13. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

14. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

15. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

16. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

17. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

18. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

19. We Found a Bug!
How did QA miss this?
That was obvious! I shouldn't Developers don't test before
have to spell out every they throw it over the wall!
detail!
Exec
Tester
Product
Manager
We did what was in
functional spec!
Business requirements
weren't clear!
BA missed it in the
technical requirements!
Dev
BA
Architect

20. Accidental Adversaries
+
Testing
+
New Bugs
- +
- +
Testing Additional Development
Software Fix
Success Tests Success
+ -
+ -
New Build
+
Development
+

21. Functional Silos
Source: http://www.danpontefract.com/images/silo.jpg

22. Functional Silos
X
Source: http://www.danpontefract.com/images/silo.jpg

23. Build it Right
Build the Right Thing
Specification By Example
Gojko Adzic, 2011 page 4

24. Build it Right
Build the Right Thing
Useless Crap
Specification By Example
Gojko Adzic, 2011 page 4

25. Build it Right
Business Failure
Build the Right Thing
Useless Crap
Specification By Example
Gojko Adzic, 2011 page 4

26. Build it Right
Business Failure
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

27. Build it Right
Business Failure Business Success
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

28. Build it Right
Business Failure Business Success
Specification By
Example
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

29. What are Specifications
By Example?

30. What are Specifications
By Example?
• Thin slices of system behaviour
• that deliver business value
• described as concrete examples
• that are potentially automatable
• without translation
• to create executable specifications
• captured in live documentation.

31. Agile Testing Quadrants

32. Specification By Example

33. Specification By Example
Business Goal

34. Specification By Example
Business Goal
Derive the scope
Scope

35. Specification By Example
Business Goal
Derive the scope
Scope
Specify collaboratively
Key Examples

36. Specification By Example
Business Goal
Derive the scope
Scope
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples

37. Specification By Example
Business Goal
Derive the scope
Scope
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples
Automate literally
Executable Specification

38. Specification By Example
Business Goal
Derive the scope
Scope
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples
Automate literally
Executable Specification
Validate frequently
Living Documentation

39. Specification By Example
Business Goal
Derive the scope
Scope
Shared Understanding
Ubiquitous Language
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples
Automate literally
Executable Specification
Validate frequently
Living Documentation

40. Source: https://docs.google.com/drawings/d/1cbfKq-KazcbMVCnRfih6zMSDBdtf90KviV7l2oxGyWM/edit
Specification By Example
Business Goal
Derive the scope
Scope
Shared Understanding
Ubiquitous Language
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples
Automate literally
Executable Specification
Validate frequently
Living Documentation

41. Derive the Scope: Story Mapping
Source: http://availagility.co.uk/wp-content/uploads/2008/10/user-story-mapping.png

42. Derive the Scope: User Stories

43. Derive the Scope: User Stories
As a _______
I want to _______
So that _______

44. Derive the Scope: User Stories
As a _______
I want to _______
So that _______
As a student
I want to purchase used books online
So that I can save money

45. Specify Collaboratively: Workshops

46. Specify Collaboratively: Workshops

47. Specify Collaboratively: Workshops
• Hold regular product backlog workshops

48. Specify Collaboratively: Workshops
• Hold regular product backlog workshops
• Full team workshops - when starting

49. Specify Collaboratively: Workshops
• Hold regular product backlog workshops
• Full team workshops - when starting
• Three amigo workshops:
• One developer
• One tester
• One analyst

50. Specify Collaboratively: Key Examples

51. Specify Collaboratively: Key Examples
Given _______
When _______
Then _______

52. Specify Collaboratively: Key Examples
Given _______
When _______
Then _______
Given “War and Peace” is available as a used book for $2.99
When Susan selects book“War and Peace”
Then “Buy used for $2.99” is displayed

53. Refining the Specification

54. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic

55. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic
• Be precise and make sure spec is testable

56. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic
• Be precise and make sure spec is testable
• Avoid “scripts” and “flows”

57. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic
• Be precise and make sure spec is testable
• Avoid “scripts” and “flows”
• Focus on business functionality not design

58. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic
• Be precise and make sure spec is testable
• Avoid “scripts” and “flows”
• Focus on business functionality not design
• Avoid UI details

59. Refining the Specification
“Specifications with examples are acceptance tests”
Gojko Adzic
• Be precise and make sure spec is testable
• Avoid “scripts” and “flows”
• Focus on business functionality not design
• Avoid UI details
• Avoid covering every possible combination

60. Refining the Specification: An Example
Free Delivery
Free delivery is offered to VIP customers once they purchase a certain number of books.
Free delivery is not offered to regular customers or VIP customers buying anything other than
books.
Customer Type Cart Contents Delivery
VIP 5 books Free, Standard
VIP 4 books Standard
Regular 10 books Standard
VIP 5 dishwashers Standard
VIP 5 books, 1 dishwasher Standard
Source: Specification by Example: How successful teams deliver the right software, Gojko Adzic, pg. 116

61. Automating Examples

62. Automating Examples
• Start small

63. Automating Examples
• Start small
• Select important examples for automation

64. Automating Examples
• Start small
• Select important examples for automation
• Plan up-front to automate

65. Automating Examples
• Start small
• Select important examples for automation
• Plan up-front to automate
• Be prepared to go slower at the start

66. Automating Examples
• Start small
• Select important examples for automation
• Plan up-front to automate
• Be prepared to go slower at the start
• Treat automation code as a first class citizen

67. Automating Examples
• Start small
• Select important examples for automation
• Plan up-front to automate
• Be prepared to go slower at the start
• Treat automation code as a first class citizen
• Avoid record and playback

68. Automating Examples
• Start small
• Select important examples for automation
• Plan up-front to automate
• Be prepared to go slower at the start
• Treat automation code as a first class citizen
• Avoid record and playback
• Avoid using pre-populated data

69. Minimize Tests Through the UI

70. Automation Tools
FitNesse Cucumber Concordian
Green
JBehave Twist
Pepper
Robot
Text Test SpecFlow
Framework

71. Validate Frequently

72. Validate Frequently
• Start with a Continuous Integration system

73. Validate Frequently
• Start with a Continuous Integration system
• Set up a Continuous Deployment system

74. Validate Frequently
• Start with a Continuous Integration system
• Set up a Continuous Deployment system
• Specify and test business logic separately
from end-to-end flows

75. Validate Frequently
• Start with a Continuous Integration system
• Set up a Continuous Deployment system
• Specify and test business logic separately
from end-to-end flows
• Organize tests along functional lines

76. Validate Frequently
• Start with a Continuous Integration system
• Set up a Continuous Deployment system
• Specify and test business logic separately
from end-to-end flows
• Organize tests along functional lines
• Run all test nightly

77. Validate Frequently
• Start with a Continuous Integration system
• Set up a Continuous Deployment system
• Specify and test business logic separately
from end-to-end flows
• Organize tests along functional lines
• Run all test nightly
• Consider an iteration “test pack”

78. Living Documentation
• Keep specifications short
• Evolve a specification language and leverage
in with “common fixtures”
• Make documentation accessible - consider
a wiki
• Organize the documentation
• Put specifications under version control

79. A Worked Example

80. Executable Specification

81. Cucumber Organization
fixtures are common
Features Step
System

82. Cucumber Organization
fixtures are common
ad
Lo
Features Step
System

83. Cucumber Organization
fixtures are common
M
ad
at
Lo
ch
Features Step
System

84. Cucumber Organization
fixtures are common
M
ad
at
Lo
ch
Features Step
Call
System

85. Feature File
Feature: Turn cucumber into beer
As a cucumber presenter
I want beer after my presentation
So I can enjoy the rest of DemoCampGuelph
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Scenario: Ali buys Declan beer
Given Ali hosts DemoCampGuelph
When Declan demos Cucumber
Then Ali should buy Declan 1 beer

86. Step Definitions
Given /^(.+) hosts/ do |host|
@event = Event.new(host)
end
When /^(.+) demos/ do |presenter|
@event.add(presenter)
end
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer, drinker,
qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker
perk.quantity.should == quantity.to_i; perk.item.should
== item
end

87. System Under Test
class Event
attr_reader :perks
def initialize(host) @host = host; @perks = [] end
def add(presenter)
@perks.push Perk.new(@host, presenter, 1, "beer")
end
end
class Perk
attr_reader :buyer, :receiver, :quantity, :item
def initialize(buyer, receiver, quantity, item)
@buyer = buyer; @receiver = receiver
@quantity = quantity; @item = item
end
end

88. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer

89. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Given /^(.+) hosts/ do | When /^(.+) demos/ do |
host| presenter|
@event = @event.add(presenter)
Event.new(host) end
End
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer,
drinker, qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker

90. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Given /^(.+) hosts/ do |
host|
@event =
Event.new(host)
End

91. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Given /^(.+) hosts/ do |
host| “Brydon”
@event =
Event.new(host)
End

92. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer

93. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Given /^(.+) hosts/ do | When /^(.+) demos/ do |
host| presenter|
@event = @event.add(presenter)
Event.new(host) end
End
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer,
drinker, qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker

94. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
When /^(.+) demos/ do |
presenter|
@event.add(presenter)
end

95. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
When /^(.+) demos/ do |
presenter|
@event.add(presenter)
end
“Declan”

96. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer

97. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Given /^(.+) hosts/ do | When /^(.+) demos/ do |
host| presenter|
@event = @event.add(presenter)
Event.new(host) end
End
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer,
drinker, qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker

98. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer,
drinker, qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker

99. Execution
Scenario: Brydon buys Declan beer
Given Brydon hosts DemoCampGuelph
When Declan demos Cucumber
Then Brydon should buy Declan 1 beer
Then /^(.+) should buy (.+) (d+) (.*)$/ do |buyer,
drinker, qty, item|
perk = @event.perks[0];
perk.buyer.should == buyer; perk.receiver.should ==
drinker

100. Source: https://docs.google.com/drawings/d/1cbfKq-KazcbMVCnRfih6zMSDBdtf90KviV7l2oxGyWM/edit
Specification By Example
Business Goal
Derive the scope
Scope
Shared Understanding
Ubiquitous Language
Specify collaboratively
Key Examples
Refine the specification
Specification With Examples
Automate literally
Executable Specification
Validate frequently
Living Documentation

101. Build it Right
Build the Right Thing
Specification By Example
Gojko Adzic, 2011 page 4

102. Build it Right
Build the Right Thing
Useless Crap
Specification By Example
Gojko Adzic, 2011 page 4

103. Build it Right
Business Failure
Build the Right Thing
Useless Crap
Specification By Example
Gojko Adzic, 2011 page 4

104. Build it Right
Business Failure
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

105. Build it Right
Business Failure Business Success
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

106. Build it Right
Business Failure Business Success
Specification By
Example
Build the Right Thing
Useless Crap Maintenance Nightmare
Specification By Example
Gojko Adzic, 2011 page 4

107. Reading
Specification By Example
Gojko Adzic
The RSpec Book: Behaviour Driven Development with
RSpec, Cucumber and Friends
David Cheliminksy et al
Agile Testing: A Practical Guide for Testers and Agile Teams
Lisa Crispin, Janet Gregory

108. Diagram Credits
Lisa Crispin and Janet Gregory
Agile Testing: A Practical Guide for Testers
and Agile Teams
Addison-Wesley Professional; January 9, 2009.
Mike Cohn
Succeeding with Agile: Software Development
Using Scrum
Addison-Wesley Professional; November 5, 2009.