Spotify uses metric alignment and A/B testing to optimize user experience across their many autonomous teams and platforms. Education and automation are key to ensuring widespread adoption of the aligned metrics. Presenting results from A/B tests can demonstrate how certain features and functions impact vital metrics like daily and monthly active users, and teams are organized as a people as a service model to best implement optimizations.

1. May 20, 2014
Managing experiments at
Spotify
Ali Sarrafi
Product Owner
A/B testing and Experimentation

6. May 20, 2014
The right music for every moment
6

7. May 20, 2014
7
24M active users.
6M paying subscribers.
20Msongs.

8. Tribes
Provide fast and
reliable access to all
the world's music
Enable high product
development speed
while maintaining a
highly available service
P P P P
Squad Squad Squad Squad
Chapter
Chapter
P P P P
Squad Squad Squad Squad
Chapter
Chapter
Our team is growing rapidly
250
30
100
10

9. 9

10. 10
Autonomous fast moving teams
Feature 1
Feature 3
Feature 2
Feature 4
Feature 5

11. May 20, 2014
Some Statistics
Over 40 autonomous teams working on features
Over 7 platforms with multiple features
Over 3000 source repositories

12. 12
We don’t want a Frankenstein software

13. 13
Well at least a good looking 2014 edition

14. Section name 14
How can we align teams’ optimisation to
avoid the frankenstein effect?

15. 15
Align on metrics rather than actual tests

16. 16
DAU (Daily Active Users)
MAU (Monthly Active Users)

17. Section name 17
Evolving Metrics

18. May 20, 2014
Focus on functionality as a metric

19. 19
DAU/MAU
Function 1 Function 2 Function 3 Function 4
CTR/Team Metrics
Importance
(High to low)
Order of
optimisation

20. Section name 20
Now we know what would work how do we
make sure everyone uses it?

21. 21
It’s all about automation

22. 22
and of course education

23. Section name 23

24. 24
Make magic happen with desired metrics
Vital Sign Group A Group B
Functionality 1 +1.5% -0.5%
Functionality 2 +2.5% -3%
Functionality 3 +0.5% 0%
Functionality 4 +1% -7%

25. 25
People/Infrastructure as a service
Feature 1
Feature 5
Feature 4
Feature 2
Feature 3

26. Summary
• Metric alignment is the most effective way of making sure the
overall experience does not suffer
• Education and Automation are key for adoption
• People as a service is a most effective way of reaching out to
teams.

27. May 20, 2014
Copyright stuff
Napster pic by bixentro, cc 2.0
http://www.flickr.com/photos/bixentro/
Busy record store pic by gomattolson
http://www.flickr.com/photos/gomattolson
Empty record store pic by passetti
http://www.flickr.com/photos/passetti/
First Frankenstein screenshot form :
“Frankenstein” movie by Kenneth Branagh, 1994
Second Frankenstein screenshot form :
“I, Frankenstein” movie by Stuart Beattie, 2014

28. May 20, 2014
Thanks
@asarrafi