When running any amount of systems, gaining visibility into what they are doing can be a non-trivial matter. Starting on the path to monitoring can prove bumpy, and if you don’t measure, you don’t know. In this session, Michael Fiedler, Director of TechOps, will speak on personal experience with scalability, deployment, and monitoring challenges prior to using Datadog - and how that changed. He will cover how to get started, and examples of where monitoring the company's platform with Datadog provided the guiding light towards the team solving scalability problems.

1. Why visibility into your
stack matters
or, Do you see it all?

2. Mike Fiedler
Operations
Datadog.com
Twitter: @mikefiedler
GitHub: @miketheman
OpsSchool.org
Chef Community
Roller Derby Referee
Skydiver
©Alex Erde

3. –CEO calling your cellphone at 03:00
“The site is slow.”

4. What?
• typical monitoring implementation story
• an alternative approach

5. (CC BY 2.0) http://www.gotcredit.com/ https://flic.kr/p/6439SA

6. LB
Data
User
Web

8. (CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

9. (CC BY 2.0) Aristocrats-hat https://flic.kr/p/6qdTC1
–W. Edwards Deming, The Elements of Statistical Learning
“In God we trust; all others bring data.”

12. You want more?

13. • graphite
• ganglia
• mongodb
• mysql
• influxdb
• socket.io
• datadog
• …

14. from bottle import route
import pymongo
import json
db = pymongo.Connection(‘mongodb://...
@route('/insert/:name')
def insert(name):
doc = {'name': name}
db.words.update(
doc, {"$inc":{"count": 1}}, upsert=True
)
return json.dumps(doc, default=default)

15. from bottle import route
import pymongo
import json
from statsd import statsd
db = pymongo.Connection(‘mongodb://...
@route(‘/insert/:name')
@statsd.increment('wordcount.insert')
def insert(name):
doc = {'name': name}
db.words.update(
doc, {"$inc":{"count": 1}}, upsert=True
)
return json.dumps(doc, default=default)

17. Time is a Cruel Master

18. (CC BY-SA 2.0)
https://www.flickr.com/theilr/
https://flic.kr/p/8MC5YM

19. Have
• systems
• applications
• services
• developers
• operators
• customers

20. Have
• systems
• applications
• services
• developers
• operators
• customers

21. Polyglot Platforms

22. Complex Systems

23. Disparate Locations

24. Information Overload

25. –CEO calling your cellphone at 03:00
“The site is slow.”

26. (CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

27. Does this matter?

28. Top-down
• work metrics
• resource metrics
• events

29. Work Metrics
throughput (rps), success/error, performance (latency)

30. Resource Metrics
utilization (%busy), saturation (queued), errors, availability

31. Events
change/build/deploy, alerts, anything notable

32. Trend resource metrics,
notify on changes

33. Wake people up when
work metrics go awry

36. Slice and Dice
exploration and aggregation

39. Set-and-Forget

41. Just-In-Time
Information

43. Does it scale?

44. Customer Stats
• AdRoll, ~2m transactions/second
• SimpleReach, ~7b measurements/day
• MercadoLibre, ~18k hosts monitored
• AirBnB, 3000+ monitors defined

45. –CEO calling your cellphone at 03:00
“The site is slow.”

46. –You
“Thanks. We know, and are already investigating.”

47. –You, because you never got that call in the first place due to
proactive data collection and alerting.
“[silence]”

49. Questions?

50. –M. Fiedler, Twitter: @mikefiedler
“If you don’t measure, you don’t won’t know.”