The Cassandra architecture shines at ensuring a very high availability of data even while nodes are failing or are overloaded. On the other hand, query latency will often rise during these events, especially on the higher percentiles. Many improvements have been made to reduce this effect over the past years. This talk will focus on one in particular: Speculative Retries. Introduced in Cassandra 2.0 on the server side and in the Java Driver 3.0 on the client side, this strategy remains complex to fully understand and to finely tune. This talk will deep dive into theoretical and practical aspects of Speculative Retries, showing the effect of tuning strategies with ad-hoc benchmarks. About the Speakers Michael Figuiere Cloud Platform Engineer, Netflix Michael is a senior software engineer at Netflix where he works on improving the cloud storage infrastructure. He previously worked at Apple and DataStax where he worked for several years on creating Drivers and Developer Tools for Cassandra. At ease with both enterprise applications and lower level technologies, he specializes in distributed architectures and topics such as databases, search engines, and cloud. Minh Do Senior Distributed Engineer, Netflix Minh Do has been working at Netflix for the last several years to run, patch, and troubleshoot Cassandra on both server and client sides, and is also a co-creator of Dynomite project. Prior to Netflix, at Tango, he spearheaded its Big Data pipeline system from the ground using Spark/Hadoop. Before that, at Qualys, he built a distributed queue system that bridges traffics between all major components. He has passion in distributed system, machine learning/deep learning, and data storages.

1. Tuning Speculative Retries
to Fight Latency
Minh Do - Senior Distributed System Engineer
Michaël Figuière - Senior Distributed System Engineer

2. Agenda
What are Speculative Retries?
C* Setting & Internal Code
C* Testing and Outcome
Datastax Java Driver

3. Speculative Retries
• “The Tail at Scale” - Google paper
• Hedged Requests vs. Speculative Retries

4. Speculative Retries
• Tuning available both in C* and Java Driver
• It is all about the additional read retries to improve
the latency tail

5. C* Setting & Internals
• Read path only
• Set at table level:
– None, Always, Custom, Percentile
• Using different executors for different behaviors
• Impacted by Read Repair (for simplicity, ignore in
this talk)

6. No Retry Policy
• Use NeverSpeculatingReadExecutor
• Coordinator computes a list of alive replicas:
– #blockFor
• Sends only one full data request to one replica
• Sends additional digest request(s) to others
depending on Consistency Level (CL)

7. No Retry Policy
Example using CL_LOCAL_QUORUM (replication factor 3)

8. Always Retry Policy
• Use AlwaysSpeculatingReadExecutor
• List of replicas: (#blockFor + 1) or all nodes
• Coordinator sends 2 full data requests to 2
replicas
• Sends additional digest requests to others
depending on Consistency Level

9. Always Retry Policy
Example using CL_ONE

10. Custom/Percentile Retry Policy
• Use SpeculatingReadExecutor
• List of replicas: #blockFor + 1
• Send 1 full data request and 1 or more digest request to
replicas depending Consistency Level
• Use the last replicas on the list for the retries
• Coordinator waits for a duration to retry on another replica
– Custom: duration in millisecs
– Percentile: the percentile on the sampling latency
distribution

11. Custom/Percentile Retry Policy
Example using CL_LOCAL_QUORUM (replication factor 3)

12. C* test - setup
• Cassandra: 12 nodes (8 cores/60Gb RAM), approx 120Gb
data per node
• Client: 6 (8 cores/30 Gb RAM)
• Use Linux Traffic Control (tc) to simulate network glitch on
one C* node using delayed packet transmission technique

13. C* test - result
• Clients send 50K reads/sec with 3K writes/sec,
• Use TC to force a slow node
• Enable 95th Speculative Retry setting
• Throughput degraded to 30K reads/sec on Coord.
• Avg. latency gets doubled 0.5ms to 1ms
• 95th/99th latencies have many spikes to 2x-10x

14. C* test - result
• Client 18K reads/sec with 1K writes/sec
• Use TC to force a slow node
• Keep 95th Speculative Retry setting on
• Throughput increased to 20+K reads/sec
• Avg. latency gets doubled 0.5ms to 1ms
• 95th/99th latencies are lower, no spikes or more
stable

15. C* test - conclusion
• More stable and lower 95th/99th latencies:
– Build cluster with extra capacities and turn on
Speculative Retry
• C* cluster is already near its max capacity
– Disable Speculative Retry as performance getting worse

16. Client Speculative Retries
• SpeculativeExecutionPolicy
• Speculative retries for both reads and writes
• But only for idempotent requests

17. Client Speculative Retries
Example using CL_LOCAL_QUORUM (replication factor 3)

18. NoSpeculativeExecutionPolicy
• Just the normal Driver behavior
• Retries are attempted only after a fixed timeout as
part of Failover

19. ConstantSpeculativeExecutionPolicy
Trigger a Speculative Retry if the Coordinator
hasn’t answer within a given delay.

20. PercentileSpeculativeExecutionPolicy
Trigger speculative retries if the chosen
coordinator doesn’t answer within a given
percentile of its typical response time

21. Questions
?