What are the most common problems with testing environments?
- You are not the only one who is using it.
- Test failures are not repeatable.
- Test data can be easily messed up due to tests overlap.
Those problems are introducing flakiness in your tests, increase frustration level and decrease confidence in quality of a product you are building. Forcing your development team to have a testing queue increases delivery time dramatically. Creating zillions of environments does not sound as cheapest solution either.
At Spotify we experimented with different approaches on how testing environments can be configured: from shared environment to mocks, stubs and hermetic servers. During my presentation I will share the lessons we learned, what worked, what not and what is the direction we are pursuing in order to stabilise our testing suites.
March 9, 2015
Software Engineer in Quality
Spotify brings you the right music for
Over 20,000 songs added every day
Started in 2006 (in Sweden)
Now 1500+ employees, 600+ engineers
5(STO, GBG, NYC, BOS, SF) engineering offices
Over 30 million songs available
Over 60 million active users
Over 15 million paying subscribers
High level architectural view
Persistent Test Environment
you are not the only one who is using it
test failures not always repeatable
test data can be easily messed up
synchronise testing efforts between teams
create environment for every team/person
test in production
Hard way of solving
test environment problem
Helios Framework (c) Spotify
Helios is a Docker orchestration platform for deploying
and managing containers across an entire fleet of
servers. Helios provides a HTTP API as well as a
command line client to interact with servers running
your containers. It also keeps history of events in your
cluster including information such as deploys, restarts
and version changes.
Not only for ‘selenium’ tests
hermetic environment for verifying back-end services
hermetic environment for verifying infrastructure jobs
hermetic environment for verifying consumer clients*
*proof of concept only
Pros vs Cons
More deterministic test
Repeatable test failures
Decreased system failure
Increased test run time
Data sets are smaller than
Consider using container technologies
Create test environment on-the-fly
Check out Helios Framework
No silver bullets. Just an alternative to Mocks and Stubs.
Spotify Engineering Culture
Check out http://www.spotify.com/jobs or
@Spotifyjobs for more information
Want to join the band?
Follow @SpotifyEng on Twitter.