Slides

1. David Poblador i Garcia // @davidpoblador
Service Availability Lead
DevOps at Spotify:
There and Back Again

2. david poblador i garcia
@davidpoblador
involved in free software since 1999
started up FS OSS consultancy company in 2000
free software in the public administration
leading infrastructure and operations teams

3. DevOps at Spotify:
from day 0

4. the beginning

5. a handful of…
engineers

6. a handful of…
systems

7. one sysadmin

8. one sysadmin
datacenters servers monitoring
on-call operating systems conf management

9. the result was…

10. a lot of technical debt…
Not
our
DC

11. but it was awesome…

12. and we moved fast…

13. Spotify interest over time (2008)

14. kept growing in all dimensions:
datacenters
employees
systems
servers

15. around 2010

16. DevOps was fading away…

17. 40ish engineers
4 people in ops
dozens of systems

18. ownership issues
teams were changing fast
priorities moving

19. Operations team not scaling

20. the result was…

21. technical debt

22. lack of proper tooling

23. 2013 and onwards

24. Spotify in numbers:
55 markets
20M+ songs
1.5 billion playlists
24M active users
6M+ paying subscribers

25. Spotify in numbers:
50+ teams building products/features
around 100 backend systems
4+ datacenters
6000+ servers

26. DevOps is back

27. every team is responsible for their
systems. end to end:
it works
it runs
it scales

28. we embed SRE engineers
in teams as needed

29. we have a core-SRE team in
charge of coordination duties:
cross cutting incidents
core on-call
…

30. teams take on-call duties

31. teams take all operational duties:
provisioning
deployment
capacity planning
incident remediation

32. operations and infrastructure
merge

33. I/O builds the Spotify platform:
alerting
backups
monitoring
data pipelines
conf management
testing environment
service ecosystem
data collection
procurement
provisioning
network
…

34. we encourage teams to build
missing pieces

35. results so far

36. results so far
20% of our teams are taking full
operational responsibility

37. results so far
the availability for the systems has not
been damaged (it has even improved)

38. results so far
we remove technical debt
faster than we create it

39. results so far
high adoption rate of our I/O platform

40. results so far
very promising results in our joint
ventures with feature teams

41. results so far
I/O teams are also taking full operational
responsibility for their systems

42. and some mistakes!

43. problems
lack of buy in
hidden work
(old ownership model)

44. Tack!
DevOps at Spotify: There and Back Again
David Poblador i Garcia
@davidpoblador
dpoblador@spotify.com