Обзор мониторинга в Grammarly, о котором я докладывал на прошлом RootConf'е. Почему мы опять решили всё изменить после перехода на докер, и как мы пришли к zipper-stack, go-carbon, carbon-c-relay (в том числе и бенчмарки альтернативных решений), как получать миллион уникальных метрик в секунду, как мы пришли к тому, что теги в условии безымянных инстансов необходимы, и как мы их сделали, как работает zipper-stack и, вообще, архитектура нашего текущего убер мониторинга.

1. МОНИТОРИНГ.
ОПЯТЬ.
Всеволод Поляков

2. Platform Engineer . Grammarly
ctrlok.com

3. Что такое метрики?

6. Успешность

7. Количество

8. Время

10. Взаимодействие

11. Внутренние процессы

12. Системные метрики

13. Зачем нужны
метрики?

14. Алерты

15. Аналитика

17. Graphite

18. Default graphite architecture

19. what?

20. what?
• RRD-like (gram.ly/gfsx)

21. what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp

22. what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)

23. what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)
• Fixed size (actually no)

24. Retention and size

25. Retention and size
• 1s:1d → 1 036 828 bytes

26. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes

27. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
whisper calc

28. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
whisper calc

29. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc

30. Retention and size

31. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes

32. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB

33. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.

34. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
• can be assign per metric

35. How
• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rocker (https://github.com/grammarly/rocker)
• rocker-compose (https://github.com/grammarly/rocker-compose)

36. Default graphite architecture

37. Default graphite architecture

38. carbon-cache.py
link

39. carbon-cache.py
• single-core
link

40. carbon-cache.py
• single-core
• many options in config file
link

41. carbon-cache.py
• single-core
• many options in config file
• default
link

42. architecture
carbon-cache.py

43. Start load testing

44. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)

45. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d

46. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf

47. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
• defaults

48. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
• defaults
• almost 1.5h to get limit :(

49. carbon-cache.py cache size → 75k ms

51. updates

52. upd time

55. results
• 75 000 ms max
• 60 000 ms flagman speed
• IO :(

56. Try to tune!
• WHISPER_SPARSE_CREATE = true
(don’t allocate space on creation)
non-linear IO load.
• CACHE_WRITE_STRATEGY =
sorted (default)

57. cache size 1k → 195k ms

58. results
• 120 000 ms flagman speed
• cache flush problem :(

59. Try to tune!
• CACHE_WRITE_STRATEGY = max
will give a strong flush preference to
frequently updated metrics and will
also reduce random file-io.

60. from 1k to 150k

61. results
• 90 000 ms flagman speed
• cache flush problem :(

62. Try to tune!
• CACHE_WRITE_STRATEGY = naive
just flush. Better with random IO.

63. from 45k to 135k

64. results
• 120 000 ms flagman speed
• still CPU

65. sorted
max
naive

67. • Maybe it’s IO EBS limitation? → 512 GB disk.

68. • Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

69. • Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

70. go-carbon
link

71. go-carbon
• multi-core single daemon
link

72. go-carbon
• multi-core single daemon
• written in golang
link

73. go-carbon
• multi-core single daemon
• written in golang
• not many options to tune :(
link

74. Start load testing

75. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)

76. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d

77. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0

78. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0

79. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
• almost 1h to get limit :(

80. 1k → 130k ms ~3k/min

81. 1k → 130k ms ~3k/min

82. 1k → 130k ms ~3k/min

85. results

86. results
• 120 000 ms flagman speed

87. results
• 120 000 ms flagman speed
• but it’s without sparse.

88. results
• 120 000 ms flagman speed
• but it’s without sparse.
• try to implement

89. try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.file.Seek(int64(remaining-1), 0)
whisper.file.Write([]byte{0})
chunkSize := 16384
zeros := make([]byte, chunkSize)
for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {
// return nil, err
// }
remaining -= chunkSize
}
if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err
}

90. Уже есть в go-carbon

91. 180 000 ms !

93. try to tune!
• max update operation = 1500

94. results
• TLDR 210 000 - 240 000 ms flagman speed
• 31 000 000 cache size!

96. try to tune!
• max update operation = 0
• input-buffer = 400 000

97. results
• 270 000 ms flagman speed
• 10-20kk cache size!

99. try to tune!
• vm.dirty_background_ratio=40
• vm.dirty_ratio=60

100. 300 000 reqs

101. results
• 300 000 ms flagman speed
• 180k+ ms ±without cache

102. Re:Lays

103. Default graphite architecture

104. Default graphite architecture

105. arch forward

106. arch namedregexp

107. arch hash

108. arch hash replicafactor: 2

109. carbon-relay.py
• twisted based
• native

110. Start load testing

111. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)

112. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan

113. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters

114. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing

115. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections

116. WTF!

117. carbon-relay-ng
link

118. carbon-relay-ng
• golang-based
link

119. carbon-relay-ng
• golang-based
• web-panel
link

120. carbon-relay-ng
• golang-based
• web-panel
• live-updates
link

121. carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
link

122. carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
• spooling
link

123. <150 000 reqs

124. carbon-c-relay
• написан на C
• advanced cluster management

125. from 100 000 to 1 600 000 reqs

126. 1 400 000 flagman speed. Or not?

127. 1 400 000 flagman speed. Or not?

128. 1 400 000 flagman speed. Or not?

129. Итак…
go-carbon + carbon-c-relay = ♡

135. Контейнеры

136. Всё перепутано

137. Различия
• Окружение
• Роль
• Трек (Модификатор)
• IP
• Датацентр
• Что-угодно

138. Теги

139. TSDB с тегами
• influxDB
• openTSDB (hbase)
• cyanite (cassandra)
• newTS (cassandra)
• Prometheus

140. (cluster) influx, 130k metrics

142. openTSDB
single instance + hbase cluster = upto 150k metrics

147. Compaction

149. Graphite

154. Найти уникальное

162. Работает с Grafana

163. Zipper
• https://github.com/grobian/carbonserver
• https://github.com/dgryski/carbonzipper
• https://github.com/dgryski/carbonapi

166. ALSO
• https://github.com/jssjr/carbonate
• https://github.com/jjneely/buckytools
• https://github.com/dgryski/carbonmem
• https://github.com/grobian/carbonwriter

167. Планы
• Патч statsd → ES
• Патч carbonserver → carbonlink

168. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club

169. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club
Мы хайрим!