The document discusses real-time monitoring tools and techniques. It introduces common tools like Graphite, StatsD, and CollectD that can be used to capture system metrics and visualize real-time performance data. The document emphasizes that real-time monitoring is important for making quick decisions, improving performance, and detecting issues promptly.

1. Real-Time-Monitoring
Max Köhler I 9. Feb. 2012
© 2012 Mayflower GmbH

2. Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 2

3. Real-Time-Monitoring

4. ... what ?

5. monitoring sucks
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 5

6. „
But does monitoring
REALLY suck?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 6

7. ...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 7

8. „
Yes, yes i know:
You can just combine Nagios + collectd + graphite +
pnp4nagios an you have everything you need!
...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 8

9. „
Seriously? Kiss my ass. I‘m a huge fan of the Unix pipeline
philosophy but, crhist, have you ever heard the phrase
„antipattern“?
- John E. Vincent
http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.html
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 9

10. https://github.com/monitoringsucks
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 10

11. monitoring sucks
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 11

12. monitoring sucks
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 11

13. monitoring sucks
is fun
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 11

14. it's all about ...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 12

15. Making Decisions
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 13

16. Making Decisions
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 14

17. How many users are using
feature xyz ?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 15

18. Feature #xyz
34%
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 16

19. Did we make it faster?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 17

20. base feature xyz
400ms
300ms
200ms
100ms
0ms
5.2 6.2 7.2 8.2 9.2
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 18

21. No !
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 19

22. Can we deploy that
for all users?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 20

23. Did we break
something?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 21

24. events / sec
60
45
30
15
0
14:00 15:00 16:00 17:00 18:00
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 22

25. Yes !
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 23

26. Will we even notice?
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 24

27. Real-Time-Monitoring I Mayflower GmbH I 9. Feb 2012 I 25

28. Changes
http://www.flickr.com/photos/addrox/2587484034/sizes/m/in/photostream/
Real-Time-Monitoring I Mayflower GmbH I 9. Feb 2012 I 26

29. Monitor!
Monitor!
Monitor!

30. „If you can not measure it,
you can not improve it!“
Lord Kelvin
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 28

31. DevOp
bingo ;)
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 29

32. Access
Yes No
http://www.flickr.com/photos/mogwai_83/3022261893/
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 30

33. Tooling
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 31

34. Carbon
StatsD
Whisper
Graphite
Logster
Real-Time
CollectD Low-Level
High-Level
Dashboards gdash
Graphiti
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 32

35. Graphite
Enterprise Scalable Realtime Graphing
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 33

36. 1. Store numeric time-series data
2. Render graphs on demand
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 34

37. Graphite
Data
rendering carbon
browser
whisper
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 35

38. StatsD
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 36

39. Graphite
statsd
rendering carbon
browser
whisper
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 37

40. Counting
StatsD::increment('events.success');
StatsD::decrement('event.count');
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 38

41. Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 39

42. Timing
$timeStart = microtime(true);
$worker->process();
$timeEnd = round(microtime(true) - $timeStart, 4);
StatsD::timing('events.timing', $timeEnd);
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 40

43. Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 41

44. Sampling
StatsD::increment('events.success', 0.1);
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 42

45. gdash
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 43

46. Dashboard
// dash.yaml
:name: Event-Queue
:description: Informationen über den Gesundheitszustand
des Patienten...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 44

47. Graphs
// timing.graph
title "Event-Queue Timing "
vtitle "Duration in s"
from "-1hours"
area :none
linewidth 1
description "Duration in s"
field :lower, :scale => 1,
:color => "green",
:alias => "Lower",
:data => "sumSeries(events.*.timing.lower)",
:derivative => false
field :mean, :scale => 1,
:color => "yellow",
:alias => "Mean",
:data => "sumSeries(events.*.timing.mean)",
:derivative => false
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 45

48. Graph Settings
title "A sample graph"
vtitle "events"
width 800
height 600
from "-2days"
until "-1days"
area :all|first|none
description "Events"
hide_legend true
linewidth 2
...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 46

49. Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 47

50. Logster
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 48

51. Log
2012/02/02 09:09:06 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
2012/02/02 14:06:07 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
2012/02/02 19:02:01 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 49

52. Log
2012/02/02 09:09:06 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
2012/02/02 14:06:07 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
2012/02/02 19:02:01 [warning] 13674#0: *24533 FastCGI sent in stderr: ...
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 50

53. Parser
....
self.reg = re.compile('magic')
....
# Return a list of metrics objects
return [
MetricObject("warning", (self.warning / self.duration), "warnings per sec"),
]
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 51

54. 600
450
300
150
0
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Real-Time-Monitoring I Mayflower GmbH I 9. Feb 2012 I 52

55. Deploy-Trend
echo "deploys 1 `date +%s`" | nc graphite.info 2003
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 53

56. 600
450
300
150
0
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Real-Time-Monitoring I Mayflower GmbH I 9. Feb 2012 I 54

57. Happy Developers
600
450
300
150
0
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Real-Time-Monitoring I Mayflower GmbH I 9. Feb 2012 I 54

58. CollectD-Graphite
Plugin
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 55

59. collectd to graphite
#collectd.conf
<Plugin write_http>
<URL "http://monitor:3012/post-collectd">
</URL>
</Plugin>
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 56

60. Start !
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 57

61. Graphite
http://gist.github.com/1023728
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 58

62. StatsD
git clone https://github.com/etsy/statsd.git
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 59

63. StatsD Client
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 60

64. Vagrant
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 61

65. Making Decisions
Low MTTD
(Mean Time To Detect)
Real-Time-Monitoring I Mayflower GmbH I 9. Feb. 2012 I 62

66. Vielen Dank für Ihre Aufmerksamkeit!
Kontakt Max Köhler
max.koehler@mayflower.de
+49 89 242054-1160
Mayflower GmbH
Mannhardtstr. 6
80538 München
© 2012 Mayflower GmbH