Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to Data at Spotify
Similar to Data at Spotify (20)
Recently uploaded
Recently uploaded (20)
Data at Spotify
- 1. June 12, 2014 Danielle Jabin Data Engineer, A/B Testing Data at Spotify
- 2. I’m Danielle Jabin • Data Engineer in the Stockholm office • A/B testing infrastructure • California born & raised • If I can survive a Swedish winter, so can you! • Studied Computer Science, Statistics, and Real Estate through the M&T program at the University of Pennsylvania
- 3. 3 Over 40 million active users As of June 9, 2014
- 4. 4 Access to more than 20 million songs As of June 9, 2014
- 5. Big Data • 40 million Monthly Active Users • 20+ million tracks • 1.5 TB of compressed data from users per day • 64 TB of data generated in Hadoop each day (including replication factor of 3) As of June 9, 2014
- 6. 6 So how much data is that?
- 7. Let’s compare: 64 TB • 293, 203, 072 books (200 pages or 240,000 characters) • 16,777,216 MP3 files (with 4MB average file size) • 22,369,600 images (with 3MB average file size)
- 8. 8 That’s a lot of selfies
- 9. 9 How do we use this data?
- 10. Use Cases • Reporting • Business Analytics • Operational Analytics • Product Features
- 11. Reporting • Reporting to labels, licensors, partners, and advertisers • We support our partners
- 12. Business Analytics • Analyzing growth, user behavior, sign-up funnels, etc • Company KPIs • NPS analysis
- 13. Operational Metrics • Root cause analysis • Latency analysis • Better capacity planning (servers, people, bandwidth)
- 14. Product Features • Discover and Radio • Top lists • Personalized recommendations • A/B Testing
- 15. 15 How do we collect this data?
- 16. The three pillars of our Data Infrastructure: Kafka Collection Hadoop Processing Databases Analytics/Visualization
- 17. This is Dave. Data Engineer at Spotify by day…
- 18. …chiptune DJ Demoscene Time Machine by night.
- 19. Let’s listen to Dave’s song
- 20. Kafka • High volume pub-sub system • “Producers publish messages to Kafka topics, and consumers subscribe to these topics and consume the messages.”
- 21. Kafka • Robust and scalable solution for collection of logs • Fast data transfer • Low CPU overhead • Built-in partitioning, replication, and fault-tolerance • Consumers can pull data at different rates • Able to handle extremely high volumes
- 22. Other people listened too!
- 23. Hadoop • Process and store massive amounts of unstructured data across a distributed cluster • One cluster with 37 nodes to 690 nodes today • 28 PB of storage • The largest Hadoop cluster in Europe
- 24. Hadoop • Entering the land of optimizations • Data retention policy • Move to JVM-based languages • MapReduce languages • Moving to Crunch, JVM-based, for speed and scalability • Python with Hadoop Streaming, Java, Hive, PIG, Scala • Sprunch: Crunch wrapper for Scala, open sourced by Spotify • Spotify open-sourced scheduler, Luigi, written in Python • Simple and easy way to chain jobs
- 25. What if we want to know more? vs
- 26. Databases • Aggregates from Hadoop put into PostgreSQL or Cassandra • Sqoop • Core data can be used and manipulated for various needs • Ad hoc queries • Dashboards
- 27. Databases • Aggregates from Hadoop put into PostgreSQL or Cassandra • Sqoop • Ad hoc queries • Dashboards
- 28. Databases • Aggregates from Hadoop put into PostgreSQL or Cassandra • Sqoop • Ad hoc queries • Dashboards
- 29. Questions?
- 30. A/B testing questions? Find me! Contr ol vs
- 31. Thank you!