Monitoring - we all have to do it, but most people don’t seem to like it very much. Etsy has been using Nagios for over a decade to monitor its infrastructure, and over that time has created a set of tools that has allowed multiple teams to deploy, manage, and scale it. In this talk we will offer guidelines on how to scale monitoring and alerting setups, ideas for workflows around monitoring, and methods of reducing friction and alert fatigue for on-call engineers.

1. Leveling Up Monitoring:
A Decade of Automating and
Scaling Nagios
Katherine Daniels and Laurie Denness
@beerops - @lozzd Velocity 2016

2. @beerops - @lozzd Velocity 2016
Katherine Daniels
@beerops
Senior Operations Engineer, Etsy
Co-Author of Effective DevOps
Laurie Denness
@lozzd
Staff Operations Engineer, Etsy
Official Graph Enthusiast

3. 3

4. Agenda
@beerops - @lozzd Velocity 2016
Automation
2
Deployinator
3
Scaling + Tooling
4
In The Beginning...
1

6. 25M
Active Buyers
About Etsy
1.6M
Active Sellers
$2.39B
2015 Annual GMS
(As of March 31, 2016)

7. Monitoring!

8. @beerops - @lozzd Velocity 2016

9. @beerops - @lozzd Velocity 2016

10. bit.ly/yaynagios

11. https://kartar.net/2015/08/monitoring-
survey-2015---tools/

13. @beerops - @lozzd Velocity 2016
In The Beginning

14. @beerops - @lozzd Velocity 2016

15. @beerops - @lozzd Velocity 2016

16. @beerops - @lozzd Velocity 2016
Sometimes your statement needs emphasis with
a black background.

17. @beerops - @lozzd Velocity 2016

18. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Templates are awesome.

19. @beerops - @lozzd Velocity 2016

20. @beerops - @lozzd Velocity 2016

21. @beerops - @lozzd Velocity 2016

22. @beerops - @lozzd Velocity 2016

23. @beerops - @lozzd Velocity 2016
define service {
use generic-service
hostgroups Linux_hosts,!email-only-servers
service_description SSH
check_command check_ssh
}

24. @beerops - @lozzd Velocity 2016
define service {
use disk-space-service
hostgroup_name email-only-servers
contact_groups ops_nonurgent
}

25. @beerops - @lozzd Velocity 2016

26. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Start small.

27. @beerops - @lozzd Velocity 2016
Nagios and Chef

28. @beerops - @lozzd Velocity 2016

29. @beerops - @lozzd Velocity 2016

30. 24

33. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Automation is awesome!

34. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Automation is awesome!
HA HA JUST KIDDING

35. @beerops - @lozzd Velocity 2016

36. @beerops - @lozzd Velocity 2016

37. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Trust but verify.

38. @beerops - @lozzd Velocity 2016
How Many Repos?

39. @beerops - @lozzd Velocity 2016

43. @beerops - @lozzd Velocity 2016

44. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
?!?!?!?!??!?!

45. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Try, fail, learn, and try again.

46. Problems

47. Problems
• Four git repos, inconsistent mess, duplication

48. Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust

49. Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique

50. Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique
• Gain confidence in changes

51. Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique
• Gain confidence in changes
• Stop editing on the production box

52. @beerops - @lozzd Velocity 2016
Nagios and Chef

53. @beerops - @lozzd Velocity 2016
Nagios and Chef
and Deployinator!

54. @beerops - @lozzd Velocity 2016
Solution 1:
Merge everything: find and remove duplication,
shared configs

55. @beerops - @lozzd Velocity 2016
Thanks Murphy!

57. @beerops - @lozzd Velocity 2016
Super Secret Option!!!

58. @beerops - @lozzd Velocity 2016

59. @beerops - @lozzd Velocity 2016

60. @beerops - @lozzd Velocity 2016

63. @beerops - @lozzd Velocity 2016
Solution 2:
Using Jenkins CI to test changes before
production

69. @beerops - @lozzd Velocity 2016
Solution 3:
Use Deployinator to run Chef recipe to generate
automated configs

72. Chart Title

73. Chart Title

74. @beerops - @lozzd Velocity 2016
Solution 4:
Use Deployinator to rsync config to all boxes

76. • git pull repo on deploy host

77. • git pull repo on deploy host
• Run Chef recipe to add automated pieces

78. • git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that

79. • git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts

80. • git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts
• Create symlink for nagios.cfg

81. • git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts
• Create symlink for nagios.cfg
• Restart Nagios

83. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Use the tools you have.

84. @beerops - @lozzd Velocity 2016
Scaling things up!

87. @beerops - @lozzd Velocity 2016

88. @beerops - @lozzd Velocity 2016

91. @beerops - @lozzd Velocity 2016

92. @beerops - @lozzd Velocity 2016

93. @beerops - @lozzd Velocity 2016
Core Workers

94. @beerops - @lozzd Velocity 2016
Core Workers

95. @beerops - @lozzd Velocity 2016

98. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
If at first you don’t succeed,
rub some webscale on it.

99. @beerops - @lozzd Velocity 2016
Iterating and Iterating

111. @beerops - @lozzd Velocity 2016
LESSONS LEARNED:
Iterate
Iterate
Iterate

112. @beerops - @lozzd Velocity 2016
To Infinity and Beyond

114. @beerops - @lozzd Velocity 2016

115. http://github.com/etsy/opsweekly

116. http://github.com/etsy/opsweekly

117. Chart Title

118. Chart Title

121. Final Lessons Learned

122. • Templates are awesome
• Start small
• Automation is awesome
• Trust but verify
• Learn from (y)our mistakes
• Iterate on the tools you have

123. Open Source Summary

124. Open Source Summary
• http://github.com/etsy/deployinator
• http://github.com/etsy/pushbot
• http://github.com/etsy/trylib
• http://github.com/etsy/opsweekly
• http://github.com/etsy/nagios-herald
• http://github.com/RJ/irccat

125. THANK YOU!
@beerops - @lozzd Velocity 2016