Given at Vancouver Python Day 2016 http://www.vanpyday.com/en/latest/index.html

1. Protect your users with
Circuit Breakers
Scott Triglia
VanPy Day 2016
1
Jim’s

2. 2
Yelp’s Mission:
Connecting people with great
local businesses.

3. Yelp Stats
As of Q1 2016
90M 3270%102M

4. Scott Triglia
4
@scott_triglia
Work with the $$$

5. Let’s talk Circuit Breakers
5

6. 6

7. 7

8. 8

9. 9

10. 10

11. Our goals today:
introduce a basic circuit breaker
11

12. Our goals today:
a modular circuit breaker
12

13. Our goals today:
test it out on several scenarios
13

14. 14

15. 15

16. 1616
1
2
3
4

17. the fundamental rule of SOA
your systems will fail
what’s your response?
17

18. 18
https://kateheddleston.com/blog/the-null-process

19. 1919
1
2
3
4

20. 20
Nygard’s circuit breaker

21. 21

22. 22

23. 23

24. 24

25. 25
Circuit Breaker States:
* Healthy (or “closed”)
* Recovering (or “half-open”)
* Unhealthy (or “open”)

26. 26

27. 27
Recovery:
* Wait for recovery_timeout seconds
* Send a trial request, trust its results

28. 28
Before a circuit breaker:

29. 29
Before a circuit breaker:
* Diners wait forever to get food

30. 30
Before a circuit breaker:
* Diners wait forever to get food
* Kitchen has a growing backlog

31. 31
Before a circuit breaker:
* Diners wait forever to get food
* Kitchen has a growing backlog
* New diners making things worse

32. 32
With a circuit breaker:

33. 33
With a circuit breaker:
* Fewer frustrated users

34. 34
With a circuit breaker:
* Fewer frustrated users
* Reduced load on the backend

35. 35
With a circuit breaker:
* Fewer frustrated users
* Reduced load on the backend
* A non-null process

36. 36

37. 37

38. 38

39. 39

40. 40

41. 41
Should our waiters all agree?
Module 1:

42. 42

43. 43

44. 44
New Behavior:
* Clients inform each other
* Processes are no longer independent

45. 45
* Propagate failure faster
* Requires distributed datastore
* Forces decisions about consistency

46. 46
What should we do in response?
Module 2:

47. 47

48. 48

49. 49

50. 50

51. 51
New Behavior:
* Code can check in advance about
healthiness of system
* Automatic monitoring!

52. 52
* Build features on top of system
health status
* Requires a single source of truth?

53. 53
Who decides we’re unhealthy?
Module 3:

54. 54

55. 55
def signal_overload(cb):
if len(jobs) > THRESH:
cb.mark_unhealthy()

56. 56
New Behavior:
* CB gets signals from anywhere
* Signal combining logic

57. 57
* Allows many (many) new signals
* Must combine signals
* Adds complexity to system

58. 58
How do we recover?
Module 4:

59. 59

60. 60
Dark launched requests:
* Reject but process normally
* Dangerous with side effects

61. 61
Synthetic Requests:
* Dark launching with fake requests
* Not necessarily representative

62. 62
Live requests:
* Definitely representative
* Exposes real users to pain

63. 63
New Behavior:
* Traffic determines health
* Removal of recovery timeouts

64. 64
* Faster(?) recovery
* No timeout tuning required
* Dark launching not always possible
* Synthetic can be unrepresentative

65. 65
in summary

66. 66
Your system will fail, have a plan!

67. 67
The basic CB is better than nothing

68. 68
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

69. 69
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

70. 70
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

71. 71
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

72. 72
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

73. 73
…and much more!

74. Much comes down to your use case
74

75. Questions?
75
striglia@yelp.com
@scott_triglia

77. 77
Can’t we do better than
rejecting requests?

78. http://techblog.netflix.com/2011/12/
making-netflix-api-more-resilient.html

79. 79
How do I safely test out a
new circuit breaker?

80. https://engineering.heroku.com/blogs/
2015-06-30-improved-production-
stability-with-circuit-breakers/