David Josephsen's presentation on Graphing Nagios. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference

1. hi.

2. hi.
dave@librato.com
@davejosephsen
github: djosephsen
Graphing

3. Graphing Nagios
dave@librato.com
@davejosephsen
github: djosephsen

4. Graphing Nagios

9. Design Patterns for Metrics
Processing

10. Things that extract,
transport and process
Things that measure Things that store and graph
All of the above

11. Today At 4:30

12. Pattern1: Centralized Polling

13. Pattern1: Centralized Polling

14. Pattern1: Centralized Polling

15. Pattern1: Centralized Polling

16. Pattern1: Centralized Polling
you ok?

17. Pattern1: Centralized Polling
you ok?
yup.

18. Pattern1: Centralized Polling
you ok?
yup.

19. Pattern1: Centralized Polling
Hey, I said you ok?

20. Pattern1: Centralized Polling
DUDE HELLO?!

21. Pattern1: Centralized Polling
host[2] is NOT OK

22. Pattern1: Centralized Polling
ya ok?

23. Pattern1: Centralized Polling
Yeah I’m ok | rta=50.582001ms; pl=0%;0
data)

24. Pattern1: Centralized Polling
(performance data)
}
Yeah I’m ok | rta=50.582001ms; pl=0%;0

25. Pattern1: Centralized Polling
all good (perfdata)

26. Pattern1: Centralized Polling
all good (perfdata)

27. Pattern1: Centralized Polling

28. Pattern1: Centralized Polling

29. Pattern1: Centralized Polling
you ok?
yup.

30. Pattern1: Centralized Polling
<——Limits Resolution

31. Pattern1: Centralized Polling
<——Limits Resolution
<——Limits Scalability

32. Pattern2: Autonomous Agents

33. Pattern2: Autonomous Agents

34. Pattern2: Autonomous Agents

35. Pattern2: Autonomous Agents

36. Pattern2: Autonomous Agents

37. Pattern2: Autonomous Agents

38. Pattern2: Autonomous Agents
Still Doesn’t Scale

39. Pattern3: The Rollup Pattern

40. Pattern3: The Rollup Pattern

41. Pattern3: The Rollup Pattern

42. Pattern3: The Rollup Pattern

43. Pattern3: The Rollup Pattern

44. Pattern3: The Rollup Pattern

45. Pattern3: The Rollup Pattern

46. Pattern3: The Rollup Pattern

47. Pattern3: The Rollup Pattern

48. Pattern3: The Rollup Pattern
gmond gmetad

49. Pattern3: The Rollup Pattern
gmond gmetad

50. Pattern3: The Rollup Pattern
gmond gmetad

51. Pattern3: The Rollup Pattern

52. Pattern3: The Rollup Pattern
Complicated, and therefore
Hard to reason about, and therefore
Hard to Maintain

53. —>

54. That’s a server right? —>

56. That’s a thread —>

57. Auto-Scaling Group 1b-17 east
instance-eaa6238ad5ff00
instance-d3b07384d11
instance-3edec49

58. instance-d3b07384d11 instance-3edec49
ZOMG IT’S 4:00 ON FRIDAY! FINISH ALL THE WORKS!!

59. That already doesn’t
exist anymore —>

64. Pattern 4: Emitter/Reporter

65. Instrumentation as code

67. output, _ := daves.function(input)

68. metric.time( output, _ := daves.function(input))

69. func handleInput(inBytes *[]byte, type string){!
!
if type == “A” {!
handleTypeA(inBytes)!
!
} else if type == “B” {!
handleTypeB(inBytes)!
!
}else if type == “C” {!
handleTypeC(inBytes)!
!
}!
}

70. func handleInput(inBytes *[]byte, type string){!
!
if type == “A” {!
handleTypeA(inBytes)!
!
} else if type == “B” {!
handleTypeB(inBytes)!
!
}else if type == “C” {!
handleTypeC(inBytes)!
!
}!
}
aCounter.Inc(1)
bCounter.Inc(1)
cCounter.Inc(1)

74. rrdtool create target.rrd !
--start 1023654125 !
--step 300 !
DS:mem:GAUGE:600:0:671744 !
RRA:AVERAGE:0.5:12:24 !
RRA:AVERAGE:0.5:288:31

76. BOB.SOMEWHERE.COM

77. THROW METRICS
HERE

78. THROW METRICS
HERE

82. http://github.com/shawn-sterling/graphios

84. Carbon Whisper Graphite

85. Carbon
2003/TCP

86. Carbon
2003/TCP
thing.cpu.load .021 1412366606
format:
<metric path>SPACE<metric value>SPACE<metric timestamp>

87. Carbon
Whisper
2003/TCP
1412366606

88. Whisper
1412366606

89. Whisper
1412366606

90. Whisper
1412366606

91. Whisper
1412366606

92. 1412366606

93. Whisper
1412366606
http://play.grafana.org

94. Woes

95. Woes

97. HTTP POST
Carbon
JSON+UDP

98. LevelDB, RocksDB, HyperLevelDB, or LMDB

99. select value from response_times
where time > '2013-08-12 23:32:01.232' and
time < '2013-08-13';

103. 1 second, 1 year
250MB

104. 1 second, 1 year
25GB
100 Measurements

105. 1 second, 1 year
2.5TB
100 Measurements
100 servers

106. 2.5TB

110. | | 2 miutes
10 Seconds
| |
| | 1 minute

111. (stored for 1 year)
1-minute resolution data
| | | |
10-second resolution data
(stored for 24 hours)

117. 1-minute, 10cents/month
5-minute, 5cents/month
Whitelist metrics here

121. Questions?