Graphite tuning story from Kyiv Devops Day 2016

1. Metrics: where and how
graphite-oriented story

2. • Vsevolod Polyakov
• Platform Engineer at Grammarly

3. Graphite
All whisper-based systems

4. Default graphite architecture

5. what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)
• Fixed size (actually no)

6. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc

7. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
• can be assign per metric

8. How
• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rocker (https://github.com/grammarly/rocker)
• rocker-compose (https://github.com/grammarly/rocker-compose)

9. Default graphite architecture

10. carbon-cache.py
• single-core
• many options in config file
• default
link

11. architecture
carbon-cache.py

12. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND, MAX_CR
• defaults
• almost 1.5h to get limit :(

13. carbon-cache.py cache size → 75k reqs

16. results
• 75 000 reqs max
• 60 000 reqs flagman speed
• IO :(

17. Try to tune!
• WHISPER_SPARSE_CREATE =
true (don’t allocate space on
creation) non-linear IO load.
• CACHE_WRITE_STRATEGY =
sorted (default)

18. cache size 1k → 195k reqs

19. results
• 120 000 reqs flagman speed
• cache flush problem :(

20. Try to tune!
• CACHE_WRITE_STRATEGY = max
will give a strong flush preference to
frequently updated metrics and will
also reduce random file-io.

21. from 1k to 150k

22. results
• 90 000 reqs flagman speed
• cache flush problem :(

23. Try to tune!
• CACHE_WRITE_STRATEGY =
naive just flush. Better with random
IO.

24. from 45k to 135k

25. results
• 120 000 reqs flagman speed
• still CPU

26. sorted
max
naive

27. • Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

28. go-carbon
• multi-core single daemon
• written in golang
• not many options to tune :(
link

29. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
• almost 1h to get limit :(

30. 1k → 130k reqs ~3k/min

32. results
• 120 000 reqs flagman speed
• but it’s without sparse.
• try to implement

33. try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.file.Seek(int64(remaining-1), 0)
whisper.file.Write([]byte{0})
chunkSize := 16384
zeros := make([]byte, chunkSize)
for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {
// return nil, err
// }
remaining -= chunkSize
}
if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err
}

34. 180 000 reqs !

36. try to tune!
• max update operation = 1500

37. results
• TLDR 210 000 - 240 000 reqs flagman speed
• 31 000 000 cache size!

39. try to tune!
• max update operation = 0
• input-buffer = 400 000

40. results
• 270 000 reqs flagman speed
• 10-20 million req cache size!

42. try to tune!
• vm.dirty_background_ratio=40
• vm.dirty_ratio=60

43. 300 000 reqs

44. results
• 300 000 reqs flagman speed
• 180k+ reqs ±without cache

45. Re:Lays

46. Default graphite architecture

47. arch forward

48. arch namedregexp

49. arch hash

50. arch hash replicafactor: 2

51. carbon-relay.py
• twisted based
• native

52. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections

53. WTF!

54. carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
• spooling
link

55. <150 000 reqs

56. carbon-c-relay
• written in C
• advanced cluster management

57. from 100 000 to 1 600 000 reqs

58. 1 400 000 flagman speed. Or not?

59. So…
go-carbon + carbon-c-relay = ♡

60. BTW. influx, 130k reqs on cluster

61. influx

62. openTSDB
single instance + hbase cluster = upto 150k reqs

63. ALSO
• zipper:
• https://github.com/grobian/carbonserver
• https://github.com/grobian/carbonwriter
• https://github.com/dgryski/carbonzipper
• https://github.com/dgryski/carbonapi
• https://github.com/dgryski/carbonmem
• https://github.com/jssjr/carbonate

64. plans
• Cyanite, retest
• newTS
• openTSDB tuninig
• zipper tuning

65. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine