3. What is SoundCloud?
• social sound platform
• record, upload, share, listen to all kinds of
sound
• Web, iPhone, Android, Mac
• 15 million users
• platform with 16k apps, over 250 in own
app store
4. Organization
• 6 locations (4 offices, main office in Berlin)
• 133 people (26 nationalities)
• 21 teams (builders, operators, pushers)
• 12 task forces, 5 work groups
11. Request flow
CDN Router
Load balancer
Cache HDFS
memcached
App / API
MySQL
Cassandra
Worker
RabbitMQ
Activities
Transcoder
12. MySQL
• main storage for all user data
• mothership: 1 Master, 9 slaves, 900GB data
• backup slave lags intentionally 1 hour behind
• automated server selection based on slave
consistency
13. Cassandra 1
activities
• index for stream events
• friend A liked track B, C uploaded track D
• write optimized (several thousands of
writes per second)
• filled by activities, read from mothership
• 2 clusters, with a data size of 1.1TB
14. Cassandra II
stitch
• time series of events in different
resolutions (hour, day, week, etc.)
• for example: plays, likes, profile views, etc.
• cluster of 16 nodes, 1TB replicated data
• data managed by own tool
• at least 50k reads/second
15. Hadoop
• storage of all raw data, logs, events, etc.
• 20 nodes, data size of 137TB
• map/reduce jobs for analytics
• initial data source for stitch
19. service architect
(oct 2011)
API requests App / API
to enrich publishs events
messages (new Comment,
Follower, etc.)
Notifications RabbitMQ
delivers push notifications
APNS Android
20. fire fighter
(nov 2011 - dez 2011)
• MySQL database optimizations
• Cassandra cluster split
• User suspension
• lot’s of performance optimizations
21. site reliability engineer
(jan 2012 - now)
• rewrite of online schema change tool
• new CI test environment
• tools to easily spawn new EC2 instances
• rewrite of master slave adapter
• ruby upgrade