Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

1. Launching PS4 with Cassandra

2. Introduction • Alexander Filipchik – Staff Software Engineer at SNEI • Dustin Pham – Staff Software Engineer at SNEI

3. Agenda • Journey towards Cassandra • Cassandra-backed PS4 Features • Ops-y Stuff • Lessons learned

4. Journey towards Cassandra

5. Challenges • Small Team • Legacy Support • Hardware Deadline • Scaling @ Peak Time

6. Why Cassandra • Strong community • Horizontally scalable architecture • Good performance • Cost effective • New adventure J 6

7. PS4 Features backed by Cassandra

8. Cassandra-backed PS4 features • What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more

21. Cassandra-backed PS4 features • What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • + more

22. Ops-y Stuff

23. Infrastructure • Hosted in cloud and physical DCs • Several hundred nodes and growing • Cluster by feature • Vnodes and Assigned token clusters • Astyanax Client

24. Stats for PS4 cloud nodes • Data throughput: Gigabytes / sec • Cassandra read/writes: > 200,000 / sec • Data size: tens of terabytes • 10M PS4 and 80M PS3 sold 24

25. Clusters • Cluster per Read/Write pattern initially • Now use cluster per feature • Seeds referenced by DNS names • Size Tiered compaction • Manual compactions for some CFs 25

26. A typical node • m2.4xl + i2.2xl • 2 ephemeral disks (~ 2 x 800 GB) • Commit log on root partition • Topology managed in the topology file managed by chef 26

27. AWS • Nodes are interleaved between AZs – Replication factor spreads data across AZ’s – Minimizes downtime due to AZ outage Availability Zone A Availability Zone C

28. Eph1 Disk Layout Eph0 Pre-Launch Launch Current ü 2 Ephemerals in a RAID 0 ü Higher throughput (io spreads into 2 devices for reading & writing) ü If you lose 1 device, you loose the array ! ü 2 Ephemerals in a RAID 1 ü Higher throughput for reading (io spreads into 2 devices), but not for writing ü If you lose 1 device, the array continues up in degraded mode. ü ½ the available space Eph0 ü 2 individual Ephemerals ü Higher throughput (io spreads into 2 devices for reading & writing) ü You lose 1 device, Cassandra stops (configurable) ü No RAID overhead Eph0 AWS m2.4xl RAID 0 Eph1 AWS m2.4xl RAID 1 Eph1 AWS m2.4xl

29. Cluster Resizing

30. Thrift Payload Size thri%_framed_transport_size_in_mb thri%_max_message_length_in_mb

31. Bouncing Nodes phi_convict_threshold

32. Inter-DC Latency

33. Monitor system health • Nagios • Kibana/Elasticsearch • Graphite • AWS Cloudwatch • App level monitoring • Opscenter

35. App level metrics

36. Lessons Learned

37. Fun with Astyanax Client • Cross DC Latencies – Several second latencies in JP and EE data centers – Astyanax configs to ensure local datacenters used • Imbalanced node traffic – Hashing algorithm (MD5 vs Murmur3) • DNS Caching in the JVM – Stale seed nodes

38. A tale of 2 Nodes

39. Cluster lessons • A single bad node can raise app latencies significantly • Taking out an entire cassandra cluster is easy (not so fun) – Compressing data before sending to cassandra helps a lot. • Corrupted SStable resulted in cascading failure

41. • Monitoring – Memtable flush frequency – Hinted handoffs – Garbage collection – Compactions – Histograms

42. • VPNs are a dangerous bottle neck • Easier to rebuild a node than to fix • Backup data – Replication factor helps but does not account for data corruption

43. • Denormalization costs • Disk is cheap but EC2s are not • TTL on almost everything • Adjust gc_grace_period based off TTL times • Transactions ? Be creative • Load test with real data

44. • Replication strategy: – Read / Write pattern – Data is source of truth or not – Data locality – User Level data vs App level data • Cluster wide commands should be staggered – Global repair L

45. Tokens • Vnodes vs Assigned Tokens – Increased chattiness on gossip protocol with vnodes – Perceived slowness on repair and cleanup operations on vnodes enabled cluster – Astyanax client does not like vnodes…

46. Compactions • Compactions are your worst enemy – larger disk usage = high cpu & longer compactions • Leveled compaction vs sized compaction – Start up time – Cpu tradeoff – IO tradeoff • Updates + Removals eat up disks

47. We are hiring… sonyentertainmentnetwork.com/careers

Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

Similar to Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra (20)

More from DataStax Academy

More from DataStax Academy (20)

Recently uploaded

Recently uploaded (20)

Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra