Best practices for monitoring your IT infrastructure using StatsD. Find dashboard examples here: https://p.datadoghq.com/sb/9b246c4ade Monitor StatsD easily with Datadog. Learn more at https://www.datadoghq.com

1. Effective
Monitoring
with

2. @alq
CTO at
Datadog

3. An application
through the naked eye

4. An application
through a monitoring
tool

5. OODA Loop (simplified)
Observe Orient
Act Decide

6. OODA Loop (simplified)
Observe Orient
Act Decide

7. OODA Loop (simplified)
Observe Orient
Act Decide
Monitorin
g
To o l

8. OODA Loop (simplified)
Observe Orient
Act Decide
Monitorin
g
To o l
Yo u

9. OODA Loop (simplified)
Observe Orient
Act Decide
Monitorin
g
To o l
Yo u
Yo u

10. OODA Loop (simplified)
Observe Orient
Act Decide
Monitorin
g
To o l
Yo u
Yo u
Yo u

11. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive

12. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive

13. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive
Else

14. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive
Garbage
In, Garbage
Out
Else

15. Timely
Initial
assumptions
Initial set of
metrics
Revised set
of metrics
Contact
with reality
Revised
assumptions

16. Timely
Initial
assumptions
Initial set of
metrics
Minutes
Not weeks
Revised set
of metrics
Contact
with reality
Revised
assumptions

17. Comprehensive
RRRRReeeesessososoououuururrcrcrcceceeesessss Work Value

18. Comprehensive
Easy to collect
generic
but not actionable
RRRRReeeesessososoououuururrcrcrcceceeesessss Work Value

19. Comprehensive
Easy to collect
generic
but not actionable
RRRRReeeesessososoououuururrcrcrcceceeesessss Work Value
Harder to collect,
custom
but most actionable

20. statsD
Easy

21. statsD
Easy
Timely

22. statsD
Easy
Timely Comprehensive

23. How statsD works
Client libraries talk to a
simple UDP server...
pageviews:100|
c@0.25
latency:320|ms
backlog:333|g
uniques:765|s
...using a simple text protocol

24. statsD types
Type Definition Example
Gauges Absolute values Queue size
Counters Per-second rates Page views
Histograms Gauge summary Page Latency
Timers Gauge distribution Page Latency
Sets Counters of unique
things Unique visitors

25. statsD problems
Type Definition Problem
Gauges Absolute values
Latest value wins.
Gauge deltas???
Counters Per-second rates
Rates, not counts (!
= rrdtool)
Histograms Gauge summary
Assumes normal
distribution
Timers Gauge distribution
Can measure much
more than time
Sets Counters of unique
things :-)

26. #1 pitfall: “Counters”
http://dtdg.co/tokyo-counters

27. How we use statsD
http://dtdg.co/tokyo-dog

28. Essential: Tagging
http://dtdg.co/tokyo-tags

29. How to get started
• statsD https://github.com/etsy/statsd
• client libraries https://github.com/etsy/statsd/wiki
(my company) 1-stop shop http://www.datadoghq.com

30. ありがとうございました。
質問？@alq
Thank you very much!
Questions？ @alq