Recommended
Recommended
More Related Content
Similar to The Anatomy of a Cascading Failure Breakdown
Similar to The Anatomy of a Cascading Failure Breakdown (20)
Recently uploaded
Recently uploaded (20)
The Anatomy of a Cascading Failure Breakdown
- 2. The Anatomy of a Cascading Failure Rareș Mușină, Tech Lead @N26 @r3sm4n
- 4. The Anatomy of a Cascading Failure
- 5. Triggering Conditions - What happened? The Anatomy of a Cascading Failure
- 6. New Rollouts Planned Changes Traffic Drains Turndowns Triggering Conditions - Change public String getCountry(String userId) { try { // Try to get latest country to avoid stale info // The update method will write the userInfo to DB UserInfo userInfo = userInfoService.update(userId); ... return userInfo.getCountry(); } catch (Exception e) { // Default to cache if service is down return getCountryFromCache(userId); } } The Anatomy of a Cascading Failure
- 7. Triggering Conditions - What happened? The Anatomy of a Cascading Failure
- 8. Triggering Conditions - Throttling The Anatomy of a Cascading Failure
- 9. Triggering Conditions - What happened? The Anatomy of a Cascading Failure
- 10. Burstiness (e.g. scheduled tasks) DDOSes Instance Death (gee, thanks Spotinst) Organic Growth Request profile changes Triggering Conditions - Entropy The Anatomy of a Cascading Failure
- 11. CPU Memory Network Disk space Threads File descriptors ………………………... Resource Starvation - Vespene Gas Is Finite The Anatomy of a Cascading Failure
- 12. Resource Starvation - Dependencies Between Resources Poorly tuned Garbage Collection Slow requests Increased CPU due to GC More in-progress requests More RAM due to queuing Less RAM for caching Lower cache hit rate More requests to backend 🔥🔥🔥 The Anatomy of a Cascading Failure
- 13. Server Overload/Meltdown/Crash/Unavailability :( CPU/Memory maxed out Health checks returning 5xx Endpoints returning 5xx Timeouts Increased load on other instances The Anatomy of a Cascading Failure
- 14. Cascading Failures - Load Redistribution The Anatomy of a Cascading Failure ELB ELB A B 500 350 100 250 ELB ELB A 600 600
- 15. Cascading Failures - Retry Amplification The Anatomy of a Cascading Failure
- 16. Cascading Failures - Latency Creep The Anatomy of a Cascading Failure
- 17. Cascading Failures - Latency Propagation The Anatomy of a Cascading Failure Service A Service B Service C ���� Service D Service E �� �� ���� ⚡ ⚡
- 18. Cascading Failures - Resource Contention During Recovery The Anatomy of a Cascading Failure
- 20. Architecture - Orchestration vs Choreography Orchestration Choreography Strategies for Improving Resilience Card service Account service User service Signup service Ship card Create Account Create user Card service Account service User service User signup event Subscribes Signup service Publishes
- 21. Capacity Planning - Do I need it? Strategies for Improving Resilience Are you operating at Google scale? You likely don’t. Go to next slide No Ok, but it will be costly and imprecise Yes
- 22. Capacity Planning - More important things Automate provisioning and deployment (🐄🐄🐄 not 🐕🐈🐹) Auto-scaling and auto-healing Robust architecture in the face of growing traffic (pub/sub helps) Agree on SLIs and SLOs and monitor them closely Strategies for Improving Resilience
- 23. “Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” Principles of Chaos Engineering Chaos Testing Strategies for Improving Resilience
- 24. Retrying - What should I retry? What makes a request retriable? ● ⚠ idempotency ● 🚫 GET with side-effects ● ✅ stateless if you can Should you retry timeouts? ● Stay tuned to the next slides Strategies for Improving Resilience
- 25. Retrying - Backing Off With Jitter Strategies for Improving Resilience
- 26. Retrying - Retry Budgets Per-request retry budget ● Each request retried at most 3x Per-client retry budget ● Retry requests = at most 10% total requests to upstream ● If > 10% of requests are failing => upstream is likely unhealthy Strategies for Improving Resilience
- 27. Throttling - Setting Timeouts Strategies for Improving Resilience Service BService A 3s timeout Service C Service D 2s timeout 5s timeout �� ��> 2s ⚠ Timeout early and be disciplined when setting timeouts ⚠ ���� ���� ⌛
- 28. Throttling - Complexity Creep Strategies for Improving Resilience ⚠ Propagate timeouts and avoid nesting ⚠ Service BService A 7s timeout Service C Service D 5s timeout 3s timeout Service FService E 3s timeout 2s timeout
- 29. Throttling - Circular Dependencies Strategies for Improving Resilience ⚠ Avoid circular dependencies at all cost ⚠ Service B Service A 5s timeout Service C 5s timeout 2s timeout3s timeout
- 30. Throttling - Rate Limiting Avoid overload by clients and set per-client limits: ● requests from one calling service can use up to x CPU seconds/time interval on the upstream ● anything above that will be throttled ● these metrics are aggregated across all instances of a calling service and upstream If this is too complicated => limit based on RPS/customer/endpoint Strategies for Improving Resilience
- 31. Throttling - Circuit Breaking Strategies for Improving Resilience Closed Open Half Open fail (threshold reached) reset timeout fail fail (under threshold) success call/raise circuit open success Service A Circuit Breaker Service B ⚠ ⚠ ⚠ ⚠ 🚫 timeout timeout timeout timeout trip circuit circuit open
- 32. gradient = (lag_noload/lag_actual) newLimit = currentLimit × gradient + queueSize Throttling - Adaptive Concurrency Limits Queue Concurrency Strategies for Improving Resilience
- 33. Fallbacks and Rejection Cache Dead letter queues for writes Return hard-coded value Empty Response (“Fail Silent”) User experience ⚠ Make sure to discuss these with your product owners ⚠ �� �� Strategies for Improving Resilience
- 34. Choosing the right tools
- 35. Tooling landscape Choosing the right tools Libraries and Frameworks Side-car proxies 🚃🚃🚃 Hype 🚃🚃🚃
- 36. Your knobs may vary Choosing the right tools ⚠ Never sacrifice observability ⚠ Side-car proxiesLibraries and Frameworks �� Simplicity of operation Simplicity of testing Simplicity of enforcement Simplicity of configuration Polyglot Predictability of results ����
- 37. Adoption Considerations - ⚠ ⚠ ⚠ Choosing the right tools
- 38. Ask Away
- 39. Thank you! 🙇 🙇 🙇