Talks by Neptune and Facebook on automated diagnosis and remediation at Scale. https://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/

1. Reducing incident MTTR
via automated diagnosis and remediation
Auto-remediation and Event Driven Automation Meetup
Kiran Gollu, Founder
Neptune.io © 2017

2. Neptune
Neptune.io © 2017 2
• One-stop incident automation-as-a-service for DevOps
• Less Alerts, More Uptime & More sleep
• Highly scalable (millions of alerts), secure, and reliable (tolerate data center
region wide outages)
Trusted by
Backed by

3. Me
Neptune.io © 2017
• Present:
• Founder, Neptune
• YCombinator and Data Collective VC backed startup
• Past:
• Founding engineer/architect at AWS S3, and DynamoDB
• Published research papers on distributed systems collaborating
with Microsoft Research

4. What I will talk about
• State of incident response today
• Alert enrichment and auto-remediation techniques / tradeoffs
• How do you solve this problem for your company?
• Demo Neptune
• Our learning's & recommendations
Neptune.io © 2017

5. What is Incident Response?
How to handle
alerts/incidents/
outages?
Many more..
Alerts
Neptune.io © 2017

6. Maturity level of incident response teams
@jpaulreed @kfinnbraun DevOps Enterprise Summit Neptune.io © 2017

7. Problem#1: Lots of alerts
7Neptune.io © 2017

8. Problem#2: Failures are complicated
Neptune.io © 2017

9. Problem#3: Debugging is hard!
9

10. Source : DevOps survey; Victor Ops incident response
#4: 95% of Time To Recovery(TTR) is still manual today
Alert Troubleshooting
Triage | Investigate | Identify
Resolution Documentation
73% 10%5% 12%
Snapshots
• Graphs & metrics
• Logs
• Webpages
Service health checks
• Internal
• External
Host/App diagnostics
• “Top”, “df –H” etc.
• Heap dumps/Stack traces
Runbooks
• On single/cluster
of hosts
• Any script, any
language
Cloud API/CLI
actions
• Start/Stop/
Reboot
• Scale up/down
Root-cause analysis
& Audit
• Heap dumps
• Logs
• Graphs
Post-mortem
• History
• Diagnostics
Neptune.io © 2017

11. FBAR : Facebook Auto Remediation Platform
“…Its doing the work of approximately 200 sys
admins…”
“We built an internal tool for AWS”
Nurse: Auto-remediation platform
“60% of problems are fixed automatically…”
Winston: Event driven automation tool
11Neptune.io © 2017

12. Your Options
Neptune.io © 2017
SaaS product, On-Premise offering on AWS,
deep integrations with monitoring tools
Open source event driven automation
Build it in house

13. Common Issues
• Noise
• non-actionable alerts
• false positives
• self-recovering alerts
• Engineer burnout: Too much manual work
• Not measuring cost of dealing with incidents
• Incorrect monitoring thresholds
• Bandaids instead of root causing problems
• Alert correlation is hard – downstream/upstream dependencies
• Not having clear incident response and escalation processeses
Neptune.io © 2017

14. Architecting automated diagnosis and remediation
Neptune.io © 2017

15. 1. One-stop incident tracking
• Helps identify those top-20% alerts causing 80% of pain
• Sorted by frequency and MTTR
• Capture:
• MTTA (mean time to acknowledgement)
• MTTR (mean time to resolution)
• Frequency of occurrence (#times a particular alert has occurred)
• Reporting + Auditing
• Audit all activity (both manual + automated)
• Leads to data-driven post mortems
Neptune.io © 2017

16. 2. Enrich Alerts
• When an alert occurs:
• Gather debugging context automatically from different tools
Use cases:
1. High memory alert
• capture top-10 memory hogs, thread dump, memory usage graphs
2. High app error rate
• capture error rate & latency trends
• app logs for 5xx errors from Splunk/Sumologic
• app health checks
Neptune.io © 2017

17. 3. Auto-Remediate repetitive alerts
• When an alert occurs:
• If it’s a known alert à Run a remediation runbook
• Use cases:
• Process crashed à restart process
• Host is unpingable à restart 3 times and escalate if still fails
• Service is down à capture graphs, run a remediation workflow
Neptune.io © 2017

18. Cultural aspects
• Have a clear incident resolution and escalation process
• Break silos : Dev / Ops can resolve and share how incidents are resolved
• Document and version your runbooks
• Single consolidated report per each incident to make post-mortems easy
• Audit all manual and automated actions for an incident
• Use your own communication tools (Slack, HipChat) but record incident logs
• Use tools to log team collaboration activity
Neptune.io © 2017

19. Neptune Demo
Neptune.io © 2017

20. How do we ensure military grade security?
Neptune.io © 2017

21. Our learning’s from 200+ SRE/Ops teams
• Automate simple things first
• Have checks in place to avoid cascading failures
• Rate limiting, handling correlated failures
• Capture state and snapshots for self-recovering alerts
• Don’t automatically fix when you don’t know root cause
• Enriching incidents is as important as automating repetitive incidents
• Availability of automation tool should be >>> your apps
Neptune.io © 2017

22. What I talked about
• Reducing MTTR gives you:
• More agility à frees up engineers to work on business critical problems
• More uptime, thus better customer experience
• More sleep, Happier engineers
• To reduce MTTR for production alerts incidents:
• create actionable alerts and fix monitoring thresholds
• Embrace automation – enrich and automate alerts
• Instill good incident response processes
Neptune.io © 2017

23. Signup for 14 day free trail!
kiran@neptune.io
Neptune.io © 2017

24. Backup
Neptune.io © 2017

25. How do you get better at it?
• Continuously eliminate manual effort involved
• Streamline your incident response workflow (cultural aspects)
• Encourage good behavior and punish bad behavior
• Measure time spent in incident response
• Make your alerts actionable
• Fix the monitoring thresholds as a continuous process
• Enrich and automate incidents to reduce MTTR
Neptune.io © 2017