Slides from my performance talk at the 2023 Cassandra summit. Here I share my tools and process for improving Cassandra's performance. We look at the OODA loop, USE method, high level observability tools and system tools such as flame graphs and bcc-tools (ebpf). Using the example of giving more memory to Cassandra, we explore how to leverage async-profiler and bcc-tools to generate cpu flame graphs and histograms of I/O performance. We can see how identifying a performance bottleneck like time spent in decompression can guide us to solving the right problems - in this case resizing compression buffers.

1. Jon Haddad, Rustyrazorblade Consulting
rustyrazorblade.com
Cassandra Performance Tuning
Like You've Been Doing It for Ten
Years

3. • DataStax
• The Last Pickle
• Committer & PMC
• Apple
• Net
fl
ix
• Out on my own!
Last 10 Years

4. Why do we care
about performance?

5. Expert Mindset
•Methodology
•Observability
•Practice

6. Methodology

8. OODA Loop

9. USE Method

10. For every resource, check
•Utilization
•Saturation
•Errors

11. utilization: the average time that the
resource was busy servicing work

12. utilization: the average time that the
resource was busy servicing work
example: cpu at 90% utilization

13. saturation: the degree to which the
resource has extra work which it can't
service, often queued

14. saturation: the degree to which the
resource has extra work which it can't
service, often queued
example: i/o device queue of 100

15. errors: the count of error events

16. errors: the count of error events
example: tcp retransmit

17. Observability

19. Anti-Patterns
• Using averages (latency) or anything
under p99
• Hiding outliers
• Averaging Summary Data
• Not understanding your tool's output

20. Anti-Patterns
• Using averages (latency) or anything
under p99
• Hiding outliers
• Averaging Summary Data
• Not understanding your tool's output

21. System Tools

22. high level: sysstat

23. # mpstat -P ALL 1 10
Linux 5.15.0-89-generic (ubuntu-vm) 12/08/2023 _aarch64_ (2 CPU)
12:21:55 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:21:56 AM all 2.16 0.00 23.24 74.59 0.00 0.00 0.00 0.00 0.00 0.00
12:21:56 AM 0 2.17 0.00 22.83 75.00 0.00 0.00 0.00 0.00 0.00 0.00
12:21:56 AM 1 2.15 0.00 23.66 74.19 0.00 0.00 0.00 0.00 0.00 0.00
12:21:56 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:21:57 AM all 3.16 0.00 25.79 71.05 0.00 0.00 0.00 0.00 0.00 0.00
12:21:57 AM 0 3.19 0.00 25.53 71.28 0.00 0.00 0.00 0.00 0.00 0.00
12:21:57 AM 1 3.12 0.00 26.04 70.83 0.00 0.00 0.00 0.00 0.00 0.00

24. ebpf / bcc-tools

25. Q: What happens if we give
Cassandra a bigger heap?

26. # cachestat-bpfcc 1 10
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
760 20159 0 3.63% 23 454
828 21858 0 3.65% 23 381
202 16282 0 1.23% 23 414
995 15487 5 6.04% 23 475
1433 12313 0 10.42% 23 523
2966 16771 0 15.03% 23 589
5002 19346 0 20.54% 23 664
6451 18185 0 26.19% 23 735
8391 18053 0 31.73% 23 806
11036 17924 0 38.11% 23 876

27. # ext4dist-bpfcc -m 1 20
Tracing ext4 operation latency... Hit Ctrl-C to end.
23:28:18:
operation = read
msecs : count distribution
0 -> 1 : 18781 |****************************************|
2 -> 3 : 15 | |
4 -> 7 : 5 | |
operation = write
msecs : count distribution
0 -> 1 : 38182 |****************************************|
2 -> 3 : 2 | |
4 -> 7 : 2 | |

28. # biolatency-bpfcc -QDmT 1 10
Tracing block device I/O... Hit Ctrl-C to end.
23:13:48
disk = b'vda'
msecs : count distribution
0 -> 1 : 10960 |****************************************|
2 -> 3 : 53 | |
4 -> 7 : 8 | |
8 -> 15 : 1 | |

29. Flame Graphs

31. compression =
{'sstable_compression': 'LZ4Compressor',
'chunk_length_kb': ?};

32. 1010101010010000111010101010101010010000111010101
0101010100100001110101010101010100100001110101010
1010101001000011101010101010101001000011101010

33. 1010101010010000111010101010101010010000111010101
0101010100100001110101010101010100100001110101010
1010101001000011101010101010101001000011101010
write to sstable

34. 1010101010010000111010101010101010010000111010101
0101010100100001110101010101010100100001110101010
1010101001000011101010101010101001000011101010
write to sstable
read from sstable

35. $ bin/nodetool tablehistograms test foo
test/foo histograms
Percentile Read Latency Write Latency SSTables Partition Size Cell Count
(micros) (micros) (bytes)
50% 785.94 379.02 1.00 35 1
75% 4055.27 454.83 1.00 35 1
95% 4055.27 454.83 1.00 35 1
98% 4055.27 454.83 1.00 35 1
99% 4055.27 454.83 1.00 35 1
Min 654.95 315.85 1.00 30 0
Max 4055.27 454.83 1.00 35 1

36. $ bin/nodetool tablehistograms test foo
test/foo histograms
Percentile Read Latency Write Latency SSTables Partition Size Cell Count
(micros) (micros) (bytes)
50% 785.94 379.02 1.00 35 1
75% 4055.27 454.83 1.00 35 1
95% 4055.27 454.83 1.00 35 1
98% 4055.27 454.83 1.00 35 1
99% 4055.27 454.83 1.00 35 1
Min 654.95 315.85 1.00 30 0
Max 4055.27 454.83 1.00 35 1

37. ALTER TABLE test.foo WITH
compression =
{'sstable_compression': 'LZ4Compressor',
'chunk_length_kb': 4};

38. Read Before Write
• Lightweight Transactions
• Counters

40. Read Ahead
More read ampli
fi
cation
Turn it down!

41. # blockdev --report
RO RA SSZ BSZ StartSec Size Device
ro 256 512 1024 0 62124032 /dev/loop0
ro 256 512 1024 0 62140416 /dev/loop1
ro 256 512 1024 0 48668672 /dev/loop2
ro 256 512 1024 0 114929664 /dev/loop3
ro 256 512 1024 0 37240832 /dev/loop4
rw 256 512 4096 0 68719476736 /dev/vda
blockdev --setra 8 /dev/vda

42. Practice

43. Set Up A Lab
Environment
• tlp-cluster
• AxonOps

44. Benchmark
• tlp-stress
• nosqlbench
• ndbench

45. • Methodology
• Observability
• Practice

46. Announcing Training!

47. Thank You!!