Presenter: Rick Branson, Infrastructure Engineer at Instagram As Instagram has scaled to over 200 million users, so has our use of Cassandra. We've built new features and rebuilt old on Cassandra, and it's become an extremely mission-critical foundation of our production infrastructure. Rick will deliver a refresh of our use cases and go deep on the technical challenges we faced during our expansion.

1. CASSANDRA @ INSTAGRAM 2014
Rick Branson, Software Engineer, Backend Engineering
@rbranson
Cassandra Summit
September 11th, 2014

2. 2013 in a Nutshell
• Moving some log-style data from Redis to
Cassandra
• Saving $, better availability, more elasticity
• Some problems, but we worked through them

3. Overview
• "The Trouble with Redis"
• A whole new category of functionality
• Useful consumer advice

4. "THE TROUBLE WITH REDIS"

5. REDIS: THE GÜD PARTS

6. The Network Part
It's a heap on the network with a great text-based protocol.

7. Extremely Efficient
It does a lot with every ounce of CPU

8. Stable
Very much so. Sets the bar for the vast NoSQL space.

9. Useful, Rich Types
You won't find anywhere else.

10. REDIS: ZDARKSIDE

11. What's in there? Hello?

12. Snapshots (BGSAVE)

13. Single-Threaded (!)

14. Memory Allocator

15. (Lack of) Durability

16. Unsafe Replication

17. THE PHILOSOPHY

18. CACHE OR DATABASE?
Neither.

19. Memory Cliff

21. Unlikely to Change :(

23. ZOOKEEPER???

24. ZOZO: *VERY* few writes,
extremely high volume, no-latency
reads without hotspots.

25. zozo agent
/var/cache/zozo
Consumer Process
Consumer Process
Consumer Process
WATCH
/zozo/files
Writer
UPDATE /zozo/files/knobs
ZooKeeper Ensemble

26. Examples
• "Knobs" key/integer live configuration
• Small (<250K) User Lists
• Security Data Sets (i.e. Blocked IPs)

27. JSON sucks.

28. Stay tuned.

29. zozo postmortem
• Agent is dead simple, never needs updating
• ZooKeeper resists both machine & human errors
• Multi-region strategy coming

30. REDIS2CASSANDRA
(omg, finally)

31. Some things we moved
• Ad Serving Metadata
• Account Suggestion Metadata
• Churn Tracking
• Follow Requests

32. Not a drop-in.

33. source target
A B
"User A has requested to follow User B"
Outbound Requests
Key Column Column
A B ...
Inbound Requests
Key Column Column
B A ...

34. INSTAGRAM DIRECT

35. Y U CASSANDRA?
• Elasticity + Scalability
• Write Availability
• Storage Efficiency
• Good Data Model Fit

36. WE CHOSE THRIFT.

37. fan-out-on-write
• Caching doesn't work
• Copy of full data for each recipient
• Atomic batches super useful

38. #nocache

40. postid
CompositeType(IntegerType(reversed=true),
IntegerType,
IntegerType)
typecode
entryid

41. POST_HEADER =>
creator_id: 12938
media_type: 1
asset_path: "/objects/23904801..."
recipients: [28910, 208192]
(SEEN, 28910)
(SEEN, 208192)
(LIKE, 208192)
(COMMENT, 9302...) =>
[19238, "George wishes he ha..."]
(COMMENT, 3499...) =>
[28910, "Weird"]

42. IMMUTABLE (mostly) LOG

43. FILTER-ON-READ

44. CQL3, please. no? ok. :(

45. THE PENDING QUEUE

48. sender
B
A C
recipients
D
Key Column Column
B [A's PostID] ...
C [A's PostID] ...
D [A's PostID] ...
D First read, delete pending

49. 2000000 (POINTER, 1000000)
1500000 (POST_HEADER, 0) => ...
(SEEN, 902394)
(SEEN, 94083)
(LIKE, 902394)
1000000 (POST_HEADER, 0) => ...
(BUMPED, 0) => 2000000
(SEEN, 1298333)
(SEEN, 848183)
...

51. Justin's Pending
25000
24000
23000
22000
21000
20000
19000
...
1234
Justin's Inbox
...
(1234, POST..)
(1234, LIKE..)
(1234, SEEN..)
(1234, SEEN..)
(1010, POST..)
(1010, SEEN..)
(1010, COMM..)
...
Inbox
...
(1234, POST..)
(1234, LIKE..)
(1234, SEEN..)
(1234, SEEN..)
(1010, POST..)
(1010, SEEN..)
(1010, COMM..)
...
Inbox
...
(1234, POST..)
(1234, LIKE..)
(1234, SEEN..)
(1234, SEEN..)
(1010, POST..)
(1010, SEEN..)
(1010, COMM..)
...
Inbox
...
(1234, POST..)
(1234, LIKE..)
(1234, SEEN..)
(1234, SEEN..)
(1010, POST..)
(1010, SEEN..)
(1010, COMM..)
...
(1234, EXCLUDE,
<JUSTIN>)

53. Rick's Unseen
849300
738939
649019
641192
629012
610392
483201
348931
Rick's Unseen
849300
738939
649019
641192
629012
610392
483201
348931
Rick's Unseen
849300
738939
649019
641192
629012
610392
483201
348931
8

54. DEPLOYMENT
• 60 x hi1.4xlarge EC2 instances
• 2X our highest estimates
• Shrunk cluster after we knew load
• Peace of mind
• Data models held together

55. We ran C* in production.
We learned some things.

56. young gc fun!

58. Yung GC has problems
• Reads = young gen garbage
• Double-collection bug in JDK 1.7+
• Best practices were for nodes with more writes

59. <blink>DEAR GOD
ALMIGHTY!!!</blink>
now we run:
10G Young Gen
20G Heap

60. This won't work out of the box.

61. -XX:+CMSScavengeBeforeRemark

62. -XX:CMSMaxAbortablePrecleanTime=60000

63. -XX:CMSWaitDuration=30000

64. -XX:+CMSEdenChunksRecordAlways
-XX:+CMSParallelInitialMarkEnabled

65. 20-core Ivy Bridge
144GB RAM
PCIe Flash
15,000 reads/sec
2,000 writes/sec

67. TWO ISSUES
• Counters. Don't use them.
• Hundreds of thousands / millions of tombstones.

68. GET
12349081
Async Storage
Port Handler
READ
12349081
Client
RPC Thread Pool
Queue
ReadStage Thread Pool
Queue

69. THANK YOU.