Anti-entropy repairs are known to be a very peculiar maintenance operation of Cassandra clusters. They are problematic mostly because of the potential of having negative impact on the cluster's performance. Another problematic aspect is the difficulty of managing the repairs of Cassandra clusters in a careful way that would prevent the negative performance impact. Based on the long-term pain we have been experiencing with managing repairs of nearly 100 Cassandra clusters, and being unable to find a solution that would meet our needs, we went ahead and developed an open-source tool, named Cassandra Reaper [1], for easy management of Cassandra repairs. Cassandra Reaper is a tool that automates the management of anti-entropy repairs of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise. I will have to cover some basics of eventual consistency mechanisms of Cassandra, after which I will be able to focus on the features of Cassandra Reaper and our six months of experience having the tool managing the repairs of our production clusters.

1. Automating Cassandra
Repairs
Radovan Zvoncek
zvo@spotify.com
github.com/spotify/cassandra-reaper
#CassandraSummit

2. About zvo

3. About zvo
Likes pancakes

4. About zvo
Likes pancakes
Does this for the 3rd time

5. About zvo
Likes pancakes
Does this for the 3rd time
Works at Spotify

6. Working at Spotify
Is autonomous
Squads responsible for their full stack
Including Cassandra

7. Cassandra

8. Node’s data
Cassandra

9. Replication
Cassandra

10. Running Cassandra
Requires many things
One of them is keeping data consistent
Otherwise it can get lost or reappear

11. Running Cassandra
Requires many things
One of them is keeping data consistent
Eventually

12. Eventual consistency
Cassandra

13. Eventual consistency
Read Repairs
Cassandra
R W

14. Eventual consistency
Hinted Handoff
Cassandra

15. Eventual consistency
Anti-entropy Repair
Cassandra

16. Coordinated process
Anti-entropy Repair

17. Coordinated process
Four steps:
Anti-entropy Repair

18. Coordinated process
Four steps:
1: Hash
Anti-entropy Repair
#
#
#

19. Coordinated process
Four steps:
1: Hash
2: Compare
Anti-entropy Repair
###

20. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream
Anti-entropy Repair

21. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream
4: Merge
Anti-entropy Repair

22. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream
4: Merge
Can go wild...
Anti-entropy Repair

23. Repair gone wild

24. Repair gone wild
Eats a lot of disk IO
● because of hashing all the data

25. Repair gone wild
Eats a lot of disk IO
Saturates the network
● because of streaming a lot of data around

26. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
● because of receiving all replicas, possibly
from all other data centers

27. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
Causes a ton of compactions
● because of having to merge the received
data

28. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
Causes a ton of compactions
… one better be careful

29. Careful repair

30. nodetool repair
Careful repair
All three intervals

31. Partitioner range
● nodetool repair -pr
Careful repair
This interval only

32. Start & end tokens
● nodetool repair -pr -st -et
Careful repair
A part of interval only

33. Requires splitting the ring into smaller intervals
Smaller intervals mean less data
Less data means fewer repairs gone wild
Careful repair

34. Smaller intervals also mean more intervals
More intervals mean more actual repairs
Repairs need to be babysat :(
Careful repair

35. The Spotify way
Feature teams meant to do features
Not waste time operating their C* clusters
Cron-ing nodetool repair is no good
● mostly due to no feedback loop
This all led to creation of the Reaper

36. The Reaper
REST(ish) service
Does a lot of JMX
Orchestrates repairs for you

37. The reaping
You:
curl http://reaper/cluster --data ‘{“seedHost” : “my.cassandra.host.net”}’
The Reaper:
● Figures out cluster info (e.g. name, partitioner)

38. The reaping
You:
curl http://reaper/repair_run --data ‘{“clusterName”: “myCluster”}’
The Reaper:
● Prepares repair intervals

39. The reaping
You:
curl -X PUT http://reaper/repair_run/42 -d state=RUNNING
The Reaper:
● Starts triggering repairs of repair intervals

40. Reaper’s features

41. Reaper’s features
Carefulness - doesn’t kill a node
● checks for node load
● backs off after repairing an interval

42. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
● because things break all the time

43. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
● multiple small intervals in parallel

44. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
Scheduling - setup things only once
● regular full-ring repairs

45. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
Scheduling - setup things only once
Persistency - state saved somewhere
● a bit of extra resilience

46. What we reaped
First repair done 2015-01-28
1,700 repairs since then, recently 90 per week
176,000 (16%) segments failed at least once
60 repair failures

47. What we reaped

48. What we reaped

49. Reaper’s Future
CASSANDRA-10070
Whatever is needed until then

50. Greatest benefit
Cassandra Reaper automates a very tedious
maintenance operation of Cassandra clusters
in a rather smart, efficient and careful manner
while requiring minimal Cassandra expertise
github.com/spotify/cassandra-reaper
#CassandraSummit

51. Thank you!
github.com/spotify/cassandra-reaper
#CassandraSummit