1/2 day Tutorial held at XP 2013 conference in Vienna Many teams struggle with the implementation of user story acceptance criteria and having a shared understanding about the expected story outcomes. This often results in missed stakeholder expectations, ad-hoc assumptions made by the team during implementation and conflict between team members and the product owner around testing. In this tutorial, you will learn how specification-by-example and acceptance test driven development will address team conflict, missed stakeholder expectations and overall increasing the level of clarity on the project end-to-end. The presentation will cover the theory behind ATDD, case-studies and practical experience from real projects and several hands-on exercises to try out the presented concepts. You will leave this tutorial with a fundamental understanding of specification-by-example and its benefits, as well as concrete pointers on how to get started using it in your own projects.

1. CHRISTIAN HASSA (CH@TECHTALK.AT)
TWITTER: @CHRISHASSA
XP 2013 Vienna, June 3rd 2013
Specification-By-Example
with Gherkin

2. About me
• Managing Partner at TechTalk
• Working as/coaching Product Owners
TechTalk office Vienna/Austria

3. • Agile consulting and delivery
• Offices: Vienna, Budapest, Zurich
TechTalk Team
About my Team

4. 4
Refining
User Stories

5. 5
Impact Mapping
Story Mapping
Specification-By-Example
Establishing a shared understanding
Why?
How?
Code
Acceptance
Criterion
Epic
Capability
Impact, Goal
Easier to define upfront Harder to define upfront
Bug
report
Isolated,
formalized
example
Feature
User Story
Example
Reminder
for a
discussion

6. 6
Collecting Acceptance Criteria
“I would try to put a book into the
shopping cart …”
“I would try to remove a book
from the shopping cart…”
“I’d check whether the shopping cart
is empty, when I enter the shop …”
“I would try to add the same book
again to the shopping cart …”
Books can be placed into
shopping cart.
Books can be removed from
shopping cart.
Shopping cart should be empty
when entering the shop.
Adding same book again to
shopping cart should increase
quantity.
As a potential customer
I want to collect books in a shopping cart
So that I can order several books at once.
“Imagine this story is
already implemented:
how would you verify it?”

7. 7
Example:
Mobile Ticket for Wiener Linien

8. 8
Exercise: Smart Alarm
• What would you try out when the following user story was
done?
• Collect and write down the list of discussed acceptance criteria
• Work in groups of 2-3
As a regular traveler
I want to setup a
smart alarm
so that I can be
warned about service
disruptions
I want to setup
multiple smart
alarms
so that I can plan
ahead for different
travel plans I have

9. 9
Using
examples

10. 10
UI wire frames,
existing UI
rules, key examples
existing artifacts,
samples
Examples for user stories

11. 11
Specification-by-Example
Examples …
• make abstract descriptions
better understandable
However …
• examples are usually not formally
exchanged or documented
Brian Marick
Examples Tests
Requirements
consist of
describe verify
fulfillment of

12. 12
Discussion of acceptance criteria
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We would like to encourage new users to
buy in our shop.
Therefore we offer 10% discount for their
first order.
Original idea for the illustration: George Dinwiddie
http://blog.gdinwidiee.com

13. 13
… illustrated with formalized examples
Given the user has not ordered yet
When the user adds a book with the price of EUR 37.5
into the shopping cart
Then the shopping cart sub-total is EUR 33.75.
Original idea for the illustration: George Dinwiddie
http://blog.gdinwidiee.com

14. 14
Discover hidden assumptions
Actually, this is not quite right:
Books on sale should be excluded.
Original idea for the illustration: George Dinwiddie
http://blog.gdinwidiee.com

15. 15
Collaboration: 3 amigos
“Happy
Path”
Technical
feasability
Exceptions,
border cases
Original idea for the illustration: George Dinwiddie
http://blog.gdinwidiee.com

16. 16
Abstract acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping
basket

17. 17
Examples for acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”

18. 18
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Examples for acceptance criteria
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
The same book can be added multiple times to the shopping basket

19. 19
The same book can be added multiple times to the shopping basket
Structure of examples
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Title: Describes intention/abstract acceptance criterion
Arrange: Context, initial state of the system
Act: Execution of the feature
Assert: Assertion of observable behaviour
And I should see the warning: “Book already existed in basket”
Triple-A
constraint
“Checks”
Chaining
up steps

20. 20
Gherkin

21. 21
Feature: Description of feature or user story
Szenariogrundriss: Beschreibung des Akzeptanzkriteriums
Szenario: Beschreibung des Akzeptanzkriteriums
Gherkin Feature Files
Background: context for all scenarios in the feature file
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>Examples: table with examples for <place holders>
Given: automated scenario steps
@tagname
@tagname
@tagname
@tagname

22. 22
Feature
Feature: Description of feature or user story
#language:en-en
@done
Feature: Shopping cart
As visitor of the web shop
I want to collect books in a shopping cart
so that I can
- prepare a short list of books I want to order
- combine multiple books in one shipment
@Tagname

23. 23
Scenario
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps
@Tagname
@inprogress
Scenario: The same book can be added multiple times to shopping cart
Given my shopping cart contains 1 copy of "Harry Potter"
When I add the book "Harry Potter" to the shopping cart
Then my shopping cart contains 2 copies of "Harry Potter"
And a warning is displayed: "You have added the same book again"

24. 24
Scenario Outline
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Examples: table with examples for <place holders>
Given/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>
@Tagname
@Tagname
Scenario Outline: Simple search
When I search for books by the phrase '<search phrase>'
Then the list of found books should contain only: <books>
Examples:
| search phrase | books | Explanation |
| Domain | 'Domain …' | whole words are matched |
| Analysis Communication | 'Bridging…', 'Analysis …' | multiple words are matched with OR |
Scenario Outline: Candidate profile can only be saved with <mandatory field>
When I enter a valid candidate profile, without filling in '<mandatory field>'
Then I see the following validation error: 'field <mandatory field> missing'
Examples: Main fields
| mandatory field |
| email |
| name |
| first name |
Examples: Extended fields
| mandatory field |
| birthdate |

25. 25
Background
Background: context for all scenarios in the feature file
Given: automated scenario steps
Background:
Given the catalog contains the following books
| Title | Author |
| Specification-By-Example | Gojko Adzic |
| Agile Testing | Lisa Crispin, Janet Gregory |
| Scrum Field Guide | Mitch Lacey |

26. 26
Scenario step arguments
Given the catalog contains the following books
| Title | Author |
| Specification-By-Example | Gojko Adzic |
| Agile Testing | Lisa Crispin, Janet Gregory |
| Scrum Field Guide | Mitch Lacey |
When I add the book "Harry Potter" to the shopping cart
Scalar arguments:
Table arguments:

27. 27
Exercise
• Work in same previous groups of 2-3
• Pick an acceptance criteria from the
previously compiled list
• Outline usage examples with the following
structure
• Given: Context, initial state
• When: utilization of the feature
• Then: assertions of observable behaviour
• Validate that scenario title (acceptance
criterion) still describes intention of
formalized scenario

28. 28
Life time of
examples

29. 29
Purpose of the examples
• Shared understanding:
acceptance criteria
• Documentation:
system details
• Regression-tests:
violated assumptions

30. 30
Continuous validation with automation
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
System
„Step Definitions“ are binding individual steps
to an automatable interface of the application.
Automatable
interface
UI
Automation
Automation does not necessarily have to bind to the UI.
Automatability of system is supported/evolving with development.

31. 31
Demo
http://www.specflow.org
Gherkin automation for .NET
• Visual Studio plugin (VS-Gallery)
• NuGet Package

32. 32
Some more complex exercises
Choose between
• Best-fare option ticket
• Smart Alarm

33. 33
Best-fare option ticket
• Assumptions
• Ticket price is always adjusted during
validation to the best option (considering
all previous trips)
• Feedback about cost of travel and ticket
option (type) after each validation
• Valid in the core zone 100
• Scope of the story
• Different travel scenarios and how the
price adjusts
• Billings resulting from the user’s travel
• Feedback to traveler (cost of travel, status
of ticket) when validating ticket
• Out of scope for this story
• Credit limit of traveler
• Authentication of ticket (validity)
As an ad-hoc traveler
I want my ticket to adjust
to the best available fare
whenever I validate
So that I don’t need to
study upfront which ticket
I need
Ticket Type Price
Single trip 2,10
24 hours 7,10
48 hours 12,40
72 hours 15,40
8 days 35,80
Week 15,00
Month 45,00
Year 365,00

34. 34
Smart Alarm
• Assumptions
• User can specify route to monitor for
disruptions
• User can specify when disruptions
should trigger the alarm earlier
(minutes before alarm)
• Scope of the story
• Different possibilities of disruptions
provided by GTFS and their impact on
activating the alarm
• Out of scope for this story
• Configuring routes (the user is
selecting an already defined route)
• Setting up alarms (the user has
already setup an alarm)
As a regular traveler
I want to have a smart
alarm that activates
earlier when there are
traffic disruptions on my
route
So that I can travel earlier
to be still on time for my
appointment

35. 35
Exercise
Choose between
• Best-fare option ticket
• Smart Alarm
Collect and discuss scenarios
• What would you try out?
• Collect examples
• Limit scope of the story

36. 36
Exercise
Refine scenarios and formalize
into Gherkin
• Start with a “warm-up” scenario
• Increase complexity with additional
scenarios
• Reconsider structure of scenarios as you
refine them
• Validate scenario title/description after
refinement

37. 37
Impact on
testing

38. 38
Test automation becomes expensive
when …
• trying to automate
manual tests
• making tests
unreadable when
automating them
• automating after
completing
development
structure
readability
point in time

39. 39
Structure
Manual tests
Asserts Multiple combined
features
Structure ACT-ASSERT-
ACT-ASSERT-
ACT-ASSERT-
…
Dependent features
Long test path with
high chance to break
Cause and impact of
error hard to trace
Automated Check
Single aspect of a
single feature
ARRANGE –
ACT –
ASSERT
Independent features
Short test path with
lower chance to break
Cause and impact of
error easy to relate

40. 40
Test automation pyramid
User
journeys
Acceptance-
criteria
Units
exploratory
testing
Source: Mike Cohn
many
few
hard
easy
Automatability

41. 41
// Go to web page 'http://localhost:40001/' using new browser instance
BrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(44, 14));
int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(34, 11));
int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(fundNr1 + 1 == fundNr2);
// Click 'Close' button
Mouse.Click(uICloseButton, new Point(26, 11));
Readability

42. 42
A readable test case
Scenario: New found items should receive a
consecutive number for the current year
Given the previous found item of the
current year had the number 145
When I register a new found item
Then the last found item of the
current year should have the number 146

43. 43
When to test (point in time)
Acceptance criteria
(ATDD, BDD)
Unit Tests
(TDD)
business view
technical view
Exploratory tests
Workflow tests
Performance, Scalability,
Usability,Security, …
definingtheproduct
criticizingtheproduct
New dimension: defining the product
Synergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick

44. 44
Manual Testing is always necessary!
User
journeys
Acceptance-
criteria
Units
exploratory
testing
Source: Mike Cohn
many
few
harder
easier
Automatability
Manual
Check
after
Story
Done
Main
success
pathes
Undiscovered
acceptance
criteria
No/(few)
manual
regression
checks
Few pathes
are enough
More time
for exploration

45. 47
Impact on
development

46. 48
Workflow - TDD

47. 49
Extending TDD for business

48. 50
Transparency for stakeholders
In Progress

49. 51
Current sprint report: all sprint scenarios

50. 52
Starting with first scenario (AC)

51. 53
Finishing the first scenario (AC)

52. 54
Progressing scenario after scenario

53. 55
Progressing scenario after scenario

54. 56
Progressing scenario after scenario

55. 57
Implementing user stories in parallel

56. 58
First user story ready for testing

57. 59
Manual testing can start even earlier

58. 60
Work done already can break again

59. 61
Work done already can break again

60. 62
See what is temporarily not working

61. 64
Living
documentation

62. 65
Living documentation
Drill into Details
(Gherkin scenarios)
Overview
(Story Map)

63. 66
User Stories vs. Features
Product/Sprint Backlog
User Story 1
AccCrit 1
AccCrit 2
User Story 2
AccCrit 3
AccCrit 4
Living Documentation
Feature 1
AccCrit 1
AccCrit 2
Feature n
AccCrit 4
AccCrit m
User Story n
AccCrit 5
AccCrit m
AccCrit 3
AccCrit 5
„Done“
• Future options of the system
• Organized/refined according to
priority, value, effort, risk, ...
• Next possible increments of
the product (units of work)
• Current state of the system
• Organized/refined for
functional overview
• Versioned and maintained
together with source code

64. 67
Lessons
learned

65. 68
Collaboration over specification
Documentation after conversation
• Collaborative discovery
• Should trigger new questions
• Infinite number of examples
• Readability and ability to automate

66. 69
Collaboration smells
• Silos and too formal hand-overs
• Formalization cannot follow implementation
• Scenarios replace communication
• Scenarios inhibit consideration of solutions
• Scenarios inhibit discovery of new things
• Duty after development is done
• Unreadable scenarios without value
• High cost for creation and maintenance
• Technical problems with automation

67. 70
Level of automation
Controller
Business Layer
Data Layer
Model
View
Browser
automation
Trigger behaviour
through controller
Assert behaviour
on model, db, ..
Setup pre-conditions
through service
interfaces
Out-of-process
In-process

68. 71
Test execution performance
• Grouping tests
• Current WIP
• Completed work
• Database
• In-memory
• Templates for setup
• Parallel execution
• Smart execution order

69. 72
Internal vs. external DSL
Example Source:
Liz Keogh
https://github.com/lunivore/tictactoe-java/blob/master/
scenarios/com/lunivore/tictactoe/scenarios/
Three_in_a_row_wins.java

70. 73
Non-functional acceptance criteria
Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms
Given 1000 users are hitting the homepage
simultaneously
Then each user should get a response within 2ms
Matt Wynne
http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing

71. 74
Tools

72. 75
Gherkin automation tools
www.cukes.info
www.behat.org
Ruby, Java, JavaScript, C++
www.specflow.org
.NET, Mono, Silverlight, WP7
PHP

73. 76

74. 77
Books
Gojko Adzic
Bridiging the
Communication Gap
Gojko Adzic
Specification by
Example
Elisabeth Hendrickson
Explore IT!

75. 78
Christian Hassa: ch@techtalk.at - @chrishassa