Слайды с доклада на РИТ++ 2016, секция рутконф.

1. МОНИТОРИНГ.
ОПЯТЬ.
Всеволод Поляков

2. Platform Engineer . Grammarly
ctrlok.com

3. Что такое метрики?

6. Успешность

7. Количество

8. Время

10. Взаимодействие

11. Внутренние процессы

12. Системные метрики

13. Зачем нужны
метрики?

14. Алерты

15. Аналитика

17. Graphite

18. Default graphite architecture

19. what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)
• Fixed size (actually no)

20. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc

21. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
• can be assign per metric

22. How
• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rocker (https://github.com/grammarly/rocker)
• rocker-compose (https://github.com/grammarly/rocker-compose)

23. Default graphite architecture

24. carbon-cache.py
• single-core
• many options in config file
• default
link

25. architecture
carbon-cache.py

26. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
• defaults
• almost 1.5h to get limit :(

27. carbon-cache.py cache size → 75k ms

30. results
• 75 000 ms max
• 60 000 ms flagman speed
• IO :(

31. Try to tune!
• WHISPER_SPARSE_CREATE = true
(don’t allocate space on creation)
non-linear IO load.
• CACHE_WRITE_STRATEGY =
sorted (default)

32. cache size 1k → 195k ms

33. results
• 120 000 ms flagman speed
• cache flush problem :(

34. Try to tune!
• CACHE_WRITE_STRATEGY = max
will give a strong flush preference to
frequently updated metrics and will
also reduce random file-io.

35. from 1k to 150k

36. results
• 90 000 ms flagman speed
• cache flush problem :(

37. Try to tune!
• CACHE_WRITE_STRATEGY = naive
just flush. Better with random IO.

38. from 45k to 135k

39. results
• 120 000 ms flagman speed
• still CPU

40. sorted
max
naive

41. • Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

42. go-carbon
• multi-core single daemon
• written in golang
• not many options to tune :(
link

43. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
• almost 1h to get limit :(

44. 1k → 130k ms ~3k/min

46. results
• 120 000 ms flagman speed
• but it’s without sparse.
• try to implement

47. try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.file.Seek(int64(remaining-1), 0)
whisper.file.Write([]byte{0})
chunkSize := 16384
zeros := make([]byte, chunkSize)
for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {
// return nil, err
// }
remaining -= chunkSize
}
if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err
}

48. Уже есть в go-carbon

49. 180 000 ms !

51. try to tune!
• max update operation = 1500

52. results
• TLDR 210 000 - 240 000 ms flagman speed
• 31 000 000 cache size!

54. try to tune!
• max update operation = 0
• input-buffer = 400 000

55. results
• 270 000 ms flagman speed
• 10-20kk cache size!

57. try to tune!
• vm.dirty_background_ratio=40
• vm.dirty_ratio=60

58. 300 000 reqs

59. results
• 300 000 ms flagman speed
• 180k+ ms ±without cache

60. Re:Lays

61. Default graphite architecture

62. arch forward

63. arch namedregexp

64. arch hash

65. arch hash replicafactor: 2

66. carbon-relay.py
• twisted based
• native

67. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections

68. WTF!

69. carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
• spooling
link

70. <150 000 reqs

71. carbon-c-relay
• написан на C
• advanced cluster management

72. from 100 000 to 1 600 000 reqs

73. 1 400 000 flagman speed. Or not?

74. Итак…
go-carbon + carbon-c-relay = ♡

80. Контейнеры

81. Всё перепутано

82. Различия
• Окружение
• Роль
• Трек (Модификатор)
• IP
• Датацентр
• Что-угодно

83. Теги

84. TSDB с тегами
• influxDB
• openTSDB (hbase)
• cyanite (cassandra)
• newTS (cassandra)
• Prometheus

85. (cluster) influx, 130k metrics
увеличить график

87. openTSDB
single instance + hbase cluster = upto 150k metrics

92. Compaction

94. Graphite

99. Найти уникальное

107. Работает с Grafana

108. Zipper
• https://github.com/grobian/carbonserver
• https://github.com/dgryski/carbonzipper
• https://github.com/dgryski/carbonapi

111. ALSO
• https://github.com/jssjr/carbonate
• https://github.com/jjneely/buckytools
• https://github.com/dgryski/carbonmem
• https://github.com/grobian/carbonwriter

112. Планы
• Патч statsd → ES
• Патч carbonserver → carbonlink

113. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club
Мы хайрим!