What are the most common problems with testing environments? - You are not the only one who is using it. - Test failures are not repeatable. - Test data can be easily messed up due to tests overlap. Those problems are introducing flakiness in your tests, increase frustration level and decrease confidence in quality of a product you are building. Forcing your development team to have a testing queue increases delivery time dramatically. Creating zillions of environments does not sound as cheapest solution either. At Spotify we experimented with different approaches on how testing environments can be configured: from shared environment to mocks, stubs and hermetic servers. During my presentation I will share the lessons we learned, what worked, what not and what is the direction we are pursuing in order to stabilise our testing suites.

1. March 9, 2015
Andrii Dzynia
Software Engineer in Quality
@adzynia
Hermetic environment
for your
functional tests

2. 2
Spotify brings you the right music for
every moment!
Over 20,000 songs added every day
Started in 2006 (in Sweden)
Now 1500+ employees, 600+ engineers
5(STO, GBG, NYC, BOS, SF) engineering offices
Over 30 million songs available
Over 60 million active users
Over 15 million paying subscribers

3. 3
https://labs.spotify.com/2013/03/15/backend-infrastructure-at-spotify/
High level architectural view

4. Persistent Test Environment
4
you are not the only one who is using it
test failures not always repeatable
test data can be easily messed up

5. 5
synchronise testing efforts between teams
create environment for every team/person
test in production
Hard way of solving
test environment problem

6. There is an easier way.
6

7. Containers.
7

8. 8
Docker @ Spotify
https://gigaom.com/2015/02/22/how-spotify-is-ahead-of-the-pack-in-using-containers/
https://blog.docker.com/2014/06/dockercon-video-docket-at-spotify-by-rohan-singh/

9. 9
Helios Framework (c) Spotify
https://github.com/spotify/helios
Helios is a Docker orchestration platform for deploying
and managing containers across an entire fleet of
servers. Helios provides a HTTP API as well as a
command line client to interact with servers running
your containers. It also keeps history of events in your
cluster including information such as deploys, restarts
and version changes.

10. High level usage example
10

11. Code
sample
11

12. DEMO
12

13. Not only for ‘selenium’ tests
13
hermetic environment for verifying back-end services
hermetic environment for verifying infrastructure jobs
hermetic environment for verifying consumer clients*
*proof of concept only

14. Pros vs Cons
More deterministic test
results
Repeatable test failures
Decreased system failure
localisation time
Increased test run time
Data sets are smaller than
in production
14

15. Alternatives
15
Stub implementations
Mock servers
https://adamcod.es/2014/05/15/test-doubles-mock-vs-stub.html

16. 16
Take Aways
Consider using container technologies
Create test environment on-the-fly
Check out Helios Framework
No silver bullets. Just an alternative to Mocks and Stubs.
https://github.com/spotify/helios

17. 17
@adzynia
adzynia@spotify.com
Spotify Engineering Culture
part 1
part 2

18. Check out http://www.spotify.com/jobs or
@Spotifyjobs for more information
Want to join the band?
Follow @SpotifyEng on Twitter.