Living with acceptance tests: Beyond Write-Once (XP NYC)
•Download as PPTX, PDF•
1 like•447 views
The document discusses strategies for maintaining acceptance tests over time beyond just writing them once. It suggests treating acceptance tests like production code by revising and refactoring them as the system evolves. Tests should describe what is being tested in a declarative style rather than specifying implementation details so their implementation can change. As a project and domain maturity, acceptance tests may move from end-to-end implementations to using the domain model directly to run faster. The test suite should also be periodically evaluated to determine if restructuring or new tests are needed.
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Similar to Living with acceptance tests: Beyond Write-Once (XP NYC)
Similar to Living with acceptance tests: Beyond Write-Once (XP NYC) (20)
Recently uploaded
Recently uploaded (20)
Living with acceptance tests: Beyond Write-Once (XP NYC)
- 1. Living with Acceptance Tests: Beyond Write-Once Daniel Wellman Twitter: @wellman E-mail: dan@danielwellman.com www.intentmedia.com XP NYC #xpnyc February 10, 2015
- 6. Activity
- 7. Goal Build the least helpful, most frustrating acceptance test suite
- 10. Revise Your Tests With the Same Care as Your Production Code
- 15. Problem: Slow and Flaky Tests
- 17. New Project End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test End-to-End Acceptance Test
- 18. How did it get this way?
- 19. No Alternatives
- 22. Run Acceptance Tests in Parallel
- 24. Failures are Hard to Diagnose
- 26. Don’t make every acceptance test run end-to-end
- 27. Ports and Adapters a.k.a. Hexagonal Architecture
- 30. Example: Fraud Protection for an Amazon-like store Example from Gojko Adzic, “Specification by Example” Given a user with no previous transaction history, And the user’s account registration country is the UK, When the user places an order with delivery country U.S., Then the transaction is marked as suspicious, But the user sees order status as “Pending.”
- 31. Before: End-to-End When(/^the user places an order with delivery country (.*),$/) do |country_name| end # ... fill_in 'Address', with: ... select country_name, from: 'Country Selector’ click_button 'Ship To This Address' expect(page).to have_content 'Your order has been placed’ # ...
- 32. After: Using Domain Model When(/^the user places an order with delivery country (.*),$/) do |country_name| end # ... country = Country.new(country_name) order = Order.new( ..., ship_to : Address.new( street: ..., country: country)) sales_clerk.accept(order) # ...
- 34. Create a Separate System Test Suite
- 36. Alternate Steps: Use Conditionals When(/^the user places an order with delivery country (.*),$/) do |country_name| if use_browser # drive a browser using Capybara; slower else # use domain objects; faster end end use_browser = ENV['USE_BROWSER_FOR_TESTS']
- 39. Write Tests Without Specifying Implementation Details
- 41. Imperative Style Given I am on the registration page, And I enter "UK" as my billing country, And I press the "Register" button, And I add an item to my cart, When I checkout And I enter "U.S." as my shipping country, And ...
- 42. Declarative Style Given a user with no previous transaction history, And the user’s account registration country is the UK, When the user places an order with delivery country U.S., Then the transaction is marked as suspicious, But the user sees order status as “Pending.” Example by Gojko Adzic, “Specification by Example” You can’t tell how these are implemented!
- 43. Use Declarative Style to Help Keep Implementation Details Out of Tests
- 44. Problem: I’m not sure what is actually tested…
- 45. How did it get this way?
- 47. Have you introduced a new domain concept?
- 48. Are you in a new stage of your product’s life?
- 50. Suggestions • Maintain your acceptance tests just like production code • Write tests which describe what, not how, so their implementations can easily evolve later • Decide when you can start moving acceptance tests from end-to-end to using the domain directly • Re-evaluate your product’s tests and testing strategy as your system grows
- 51. Resources
- 52. Photo Credits “HighLine_061409_2779” by Jessica Sheridan is licensed under CC BY 2.0 “Highline New York” by Rebecca Krebs is licensed under CC BY 2.0 “NYC: The Highline" by Wally Gobetz is licensed under CC BY-NC-ND 2.0 “Rot”, “Yellow Cart”, and “Many Hands Make Light Work” by Oliver Rich are licensed under CC BY-NC-ND 2.0 “highline-1010384.jpg” by keroism is licensed under CC BY-SA 2.0 “highline” by Elijah Porter is licensed under CC BY-NC-SA 2.0 “Highline_030” by Timothy Vogel is licensed under CC BY-NC 2.0 “201106073451_DSC_0014_1.JPG” by Leonel Ponce is licensed under CC BY-NC 2.0 “193” by verndogs is licensed under CC BY-NC 2.0
- 53. Living with Acceptance Tests: Beyond Write-Once Daniel Wellman Twitter: @wellman E-mail: dan@danielwellman.com www.intentmedia.com XP NYC #xpnyc February 10, 2015
Editor's Notes
- My name is Daniel Wellman and I’m a software engineer at Intent Media here in New York. I worked for eight years as a software developer building projects using Test-Driven Development, and coached teams in learning and applying TDD. In that time, I’ve learned some lessons about writing acceptance tests - and some ideas I’m trying out that I’d like to talk about today.
- This is the High Line Park in New York. If you haven’t had a chance to see it, I highly recommend you check it out. It’s a park on the West Side of New York, 30 feet above the ground.
- The High Line was originally an elevated rail track built in the 1930s. It allowed freight trains to unload directly in the busy factories and warehouses of the West Side. But after nearly 50 years of service, the last train ran in 1980.
- The railway sat abandoned for twenty years. Grasses and plants, left unchecked, overtook the rails. The steel rusted away. Many people called it an eyesore, and it was scheduled for demolition. But a group saw potential and wanted to save it.
- And with a lot of care, the old, abandoned railway become something new. I chose to share this with you because I think there are similar themes of growth, renewal, and maintenance that we can apply to our software test suites. .. AND it’s a nice park and I’d encourage you to go!
- But first - an activity to start our evening out! Imagine you’re on a team and your goal is to build the least helpful, most frustrating acceptance test suite possible. How would you do that? —>
- Although I’ve not set out to build a test suite like this, sometimes it happens anyways. So how do the tests get this way, and how do we improve them?
- Many teams have write-once tests; that create them as part of a story, get them to pass, and then never touch them again. Or maybe they add additional conditions and clauses on to that test.
- If there’s one theme to this talk that I’d ask you to take away is to treat your tests with just as much care as you would your production code.
- Left untended, your acceptance tests can grow out of control. If they are written well, they may continue to pass and stay green as your system evolves, and you may never notice as their runtimes creep up longer and longer. But you can practice the same activities you do with your production code:
- … “Is this feature that I’m testing tested somewhere else?”
- Have I learned something new about my domain that might suggest a restructuring or dividing of responsibilities?
- Measuring test runtime is one easy metric to track. I’ve heard of teams that set limits on their build times, and call a point of order when the tests get to that threshhold. Then they must decide to speed up the tests or allow the runtime to grow to another checkpoint.
- Slow tests are a problem because … Flaky tests are a problem because …
- End-to-End Tests The way I like to start a project is with an end-to-end test… as a user would use it.
- When I start a system, I’ll start with an end-to-end test to help me learn about the external components I need to interact with. I use it to help drive down risk, and I’m willing to pay the price up front of building a thorough end-to-end test because it helps get confidence that I understand how all the pieces might talk to each other. But these tests can grow rapidly and slowly cause build times to creep up.
- Let’s do some root cause analysis!
- No other alternative besides end-to-end / People copy patterns (but need examples of other ways to test beyond end-to-end test)
- That is, when you find a failure, make an end-to-end test. And repeat every single time. There are other ways to deal with this problem — we’ll come back to this.
- Acceptance tests written to explain user experience
- This is the most common solution I see teams take; and often stop there. … but maybe I’m viewing projects at a similar point in time!
- For a little while, this can work…. … If you have a 45 minute build and run it in parallel to get 10 minutes, you still have the risk of the 45 minute build - it just runs in 10 minutes. Problems: Same flakiness, risk of failure but now more frequent! Susceptible to many environment issues like network, timing, etc.
- End-to-end tests help you learn that something failed, but because so much is being tested, it can be hard to get diagnostics telling you why it failed. So when we parallelize, our tests run faster, but our ability to diagnose failures does not get faster.
- So how might we deal with the flaky tests?
- Skip?
- Maybe, but you can view acceptance tests as living documentation of how your system works. If you delete them, you won’t have a complete picture.
- At some point in our project’s lifecycle, we may have confidence that our end-to-end acceptance tests are giving us the safety we need. At this point, we can switch to start writing tests that use the domain directly. But how can we ensure we aren’t missing any tests?
- The slow and flaky parts of your test tend to be the external integration points. So why not isolate those parts of the system, and in the process focus on the clarity of your application’s domain?
- Separate your application into the inside (the domain) and the outside (technical infrastructure). This has the nice side effect of pushing domain logic into the center that might have been only covered by tests.
- Meta: Tim said this was when squeezing your tests into the domain, it helps you discover when you have business logic in your adapter layer / in the UI.
- We’re shipping orders that customers did not place. We need to detect when a transaction is likely to be fraudulent so that we can stop shipment to investigate to see if this order is legitimate.
- … focused on interacting with the UI elements …
- You’d call the objects in your domain that your adapter layers would call to do their work.
- THEREFORE: This means you can replace some of your slow end-to-end tests with fast ones. (but you’ll still want some!) * Take a breath!!! *
- Manage your risk by splitting out a separate suite just for testing system integrations. Not necessarily using the same tools you use for your acceptance test suites Might only be for the programmers Probably only check happy paths
- In general, I recommend a balance of tests something like this. Remember, your acceptance tests are not necessarily end-to-end! Jez Humble’s “Journey Tests” as an alternative to story-level acceptance tests.
- … use an environment variable, and you could use an ‘if’ statement … … or something more clever like a factory. See ‘The Cucumber Book for Java’ for examples of how to do this.
- … * Take a breath *
- “… But all the tests interact with the UI!” If that’s the case, change how you write the tests.
- That is, no steps involving a GUI, web services, etc.
- “So let’s take a moment to read this example…” *pause* Problem is if you want to replace the implementation, you can’t. You’re stuck talking about the GUI. Explains how, not what. UI interactions, procedural steps, etc. Note: Not always bad, sometimes the focus *is* the interaction!
- “Now let’s take a moment to read this other example…” *pause* “What you want to have happen, now how you want it to happen.” … you read this test, and you don’t know how it’s implemented - it could be interacting with a web browser, or a REST web service, or directly with the domain objects. This gives you the freedom to change your mind later about how you want to test it.
- And the benefit of doing this is that you get the freedom later on in your project to change your mind how you want to implement those tests. Which means you have freedom to replace with fast or slow implementations. * Take a breath *
- Pretend you are describing a human workflow (not systems) Helps you focus on the domain language
- Let’s look at a positive spin… this could be a sign of a successful product that is evolving. Product Lifecycle (Simple IM Story here?)
- Technical So are there examples of other ways to implement or write tests?
- … just like you would with your production code. Spring Cutback at High Line. They do this at the beginning of Spring every year. And they don’t throw away the plants - they trim them back so they can grow again. … What would that look like on your project?
- Product Cognitive dissonance (“It says this but we all know it really means …”) (Example???) … In DDD, you may be discovering new terms in your ubiquitous language.
- Product Things may have changed and your tests may need revisiting. “Where does this feature belong?” “Who do I talk to about this feature?” “Is this feature still relevant now?”
- IM Slurp Story — Big product, one team -> Little systems sprouted out to support the major business function, then over time the little systems become important on their own.
- * Pause to let people take a photo *
- Thank you!