Zero to tested
•Download as PPTX, PDF•
0 likes•586 views
Matt Robbins Presentation on "Zero to Tested with Docker and Ruby"
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Zero to tested
Similar to Zero to tested (20)
More from MagenTys
More from MagenTys (8)
Recently uploaded
Recently uploaded (20)
Zero to tested
- 1. Zero to Tested “Bullet proof test automation in hostile environments.”
- 2. The test automation promised land Small team of full stack developers (with specialists in specific areas) BDD/TDD Strong adherence to the Test Pyramid Isolation of different layers (good use of stubs, mocks, fakes) Thin layer of e2e tests
- 3. For most projects Fairly unrealistic, at least not anytime soon...
- 4. But this means... “Often reliant on end to end automated tests which have a terrible reputation!”
- 5. Why? People acknowledge e2e tests are essential Tests are flaky CI infrastructure is poorly managed (configuration drift) Development of tests lags behind the development cycle (“test sprints”) Often ignored by most of the team
- 7. But I prefer https://www.symphonious.net/2015/04/30/making-end-to-end-tests-work/ 10,000 end to end acceptance tests, primary form of quality control complete in 50 minutes, including deploying/starting and tearing down the servers all required to pass before we consider a version releasable included in our information radiators to ensure the team has constant visibility are owned by the whole team
- 8. My theory. Us “test automation” folk have let ourselves down Even if we are not in an agile utopia… Have we exhausted every avenue to make tests reliable? Need to prove we are doing everything we can to write bulletproof tests
- 9. Now for the practical stuff! Even with minimal influence on budget/project direction we can usually hugely improve on: CI Infrastructure automation Automation code and architecture Let’s take a look at a thin vertical slice through a solid automation setup...
- 11. Infrastructure automation Start from an empty EC2 account and finish with: Fully functioning Jenkins instance Automated tests running successfully against a “flaky” test application Deployed in < 5 mins, very little code, fully automated and “Ephemeral”
- 12. DevOps Toys - Chef, Packer, Terraform, AWS, Docker BAKE SPIN UP RUN TESTS
- 13. Chef
- 14. Packer
- 15. DevOps Toys - Chef, Packer, Terraform, AWS, Docker BAKE SPIN UP RUN TESTS
- 16. Terraform
- 17. DevOps Toys - Chef, Packer, Terraform, AWS, Docker BAKE SPIN UP RUN TESTS
- 19. Docker
- 20. Docker
- 21. Has it finished?
- 22. Recap... Automated provisioning of AWS infrastructure Automated build of ‘pre-baked’ AMI Automatically provisioned Jenkins with Docker Docker allows clients to have complete control over dependencies Job DSL for building jobs
- 23. Take away Everything is in source control, no more snowflake CI boxes, no more handcrafted Jenkins jobs. “But it’s not scalable!” - switch to agent model with auto scaled agents “But it’s not secure!”- not at the moment, but easy to add e.g. github auth More importantly It’s ephemeral - you can destroy if compromised
- 24. Automation Code & Architecture
- 25. Tooling choice In Ruby Land the “go-to” libraries are: Capybara Selenium Webdriver Poltergeist/PhantomJS SitePrism
- 26. Automation Code What are the biggest failings of most end to end regression test suites?
- 27. My thoughts Inappropriate synchronisation strategies (page load, async JS) Inability to deal with non-deterministic applications and environments Inability to deal with “eventual consistency” Poor reporting of failure when it occurs
- 28. RETRY ALL THE THINGS!
- 29. A flaky app...
- 30. Page Load Handled by Selenium Webdriver!
- 32. Async JS Good libraries have first class support for this:
- 33. Visibility Elements exist in DOM but CSS means they are not visible. Good libraries have first class support for this:
- 34. Visibility Worth remembering… ● display:none ● visibility:hidden ● Opacity:0 ● Dimensions And remember there is a W3C spec for Webdriver: https://w3c.github.io/webdriver/webdriver-spec.html#dfn-element-displayed
- 35. Non determinism Should we be expected to deal with this? Maybe, maybe not Either way don’t just give up, at least detect, log and report on it!
- 36. Non determinism Don’t just rely on re-run - be proactive and report on failure Libraries probably won’t help here!
- 37. Eventual Consistency Caching, load balancing, asynchronous architectures etc. TL;DR - You might need to wait a bit...
- 38. Eventual Consistency You might have to ‘roll your own’ wait helpers from time to time
- 39. Logging Configurable levels: debug, info, warn, error etc Pretty colours - makes debugging broken builds easier Have log aggregation for your application? (RSyslog, Logstash, Kibana) Now you can combine test and application logs for super debugging power!
- 40. Test the tests We should unit test any “library code” that has significant complexity.
- 41. Static Analysis Helps maintain consistency, best practice and readability of code.
- 42. Reporting
- 43. Reporting
- 44. Wrapping up We need to up our game when writing end to end tests Make the provision of CI infrastructure automated and ephemeral Write automation code that can handle hostile conditions Report on your findings to effect change
- 46. Questions?
Editor's Notes
- Hopefully we will look at some cool devops tools and automation techniques on the way, but first some context.
- Show empty EC2 account and kick off `terraform apply`
- Get the url of the instance and show that we have a seed job and that the seed job has created the main job. Start the main job if it hasn’t already - explain that first run will be slow as it builds docker container for the first time and gets dependencies over the internet.
- Demo the endpoints in the browser.
- Demo entire CI job.
- Have a look at the REPO - point out the structure and README - starting locally should just be as simple as having Vagrant and doing `vagrant up`