This document provides an overview of site reliability engineering (SRE). It discusses that SREs work to keep sites up, know the production environment, and help build infrastructure for monitoring, deployment, and automation. Hiring SREs can help improve uptime and utilize their experience from similar systems at scale. SREs should be involved in discussions affecting the production environment and help make software more reliable and fault-tolerant.

1. SRE FROM SCRATCH

2. SITE RELIABILITY
ENGINEERING

3. PRODUCTION ENGINEERING

4. DEVOPS?

5. WHAT DO SRE DO?

6. KEEPTHE SITE UP

7. KNOWTHE PRODUCTION
ENVIRONMENT

8. KNOWTHEIR PRODUCT

9. LIAISON, ADVISOR,
CONSULTANT

10. TOOLING AND AUTOMATION

11. TRIAGE

12. SO? WHY DO I NEEDTHEM?

13. UPTIME

14. THE ENVIRONMENT IS A
PRODUCT

15. THEY’VE DONETHIS BEFORE

16. OK... LET’S HIRE SOME

17. WHATTO LOOK FOR...

18. SRES!

19. SYSADMINSTHAT PROGRAM

20. PROGRAMMERSTHAT DO
SYSADMIN

21. EXPERIENCE WITH SCALE

22. HOW DO I INTERVIEWTHEM?

23. FUNDAMENTALS

24. HARDWARE

25. SYSTEM INTERNALS

26. UNIX ENVIRONMENT

27. NETWORKING

28. APPLICATION SUPPORT

29. OPERATING AT SCALE

30. PROGRAMMING

31. DON’T HIRE HEROES

32. OK, I’VE HIRED SOME,WHAT
SHOULDTHEY DO?

33. DESIGN REVIEW

34. DATA FLOWS

35. DEPENDENCIES

36. FAILURE CONDITIONS

37. SCALING

38. LAUNCH PREPAREDNESS

39. DOCUMENTATION

40. BUILD INFRASTRUCTURE

41. MONITORING

42. DEPLOYMENT

43. OPERATORTOOLS

44. CONFIGURATION
MANAGEMENT

45. SELF-SERVICE

46. HOW SHOULDTHETEAMS
INTERACT...

47. DON’T GIVE ALLTHE DAY-TO-
DAYTASKSTOTHE SRES

48. SHARETHE LOAD

49. HAVEYOUR SRES SIT WITH
YOU

50. INCLUDETHEM IN
DISCUSSIONSTHE AFFECTTHE
PRODUCTION ENVIRONMENT

51. SOFTWARE IS NEVER
THROWN OVERTHE WALL

52. HAND-OFFS

53. SRES SHOULD BLOCK
DANGEROUS CHANGES

54. IFYOUR SRES ARE FIGHTING
FIRES,THEY’RE NOT BUILDING
INFRASTRUCTURE

55. IFYOUR SOFTWARE IS
CAUSING FIRES, FIX IT

56. ASKYOUR SRETO HELP MAKE
FLAME-PROOF SOFTWARE

57. DON’T HIDEYOUR PROBLEMS
FROM SRE

58. SRE SHOULD BE INVOLVEDTO
UNDERSTANDTHE PROBLEM

59. EVERYONE SHOULD BE
WRITING CODE OR MAKING
HARD DECISIONS

60. OF COURSETHERE ARE
OPTIONS...

61. SRE CAN DO ALLTHE
SUPPORT

62. SRES ARE A LIMITED
RESOURCE

63. SWE CAN SUPPORT
PRODUCTS...

64. APP SUPPORT BY SWE,
INFRASTRUCTURE SUPPORT
BY SRE

65. OR JUST ROTATE AROUND

66. ANY PRODUCTION ADVICE?

67. SELF-SERVICE

68. ALLTOOLS SHOULD BE
WRITTEN WITHTHE IDEATHAT
ROBOTS CAN RUNTHEM

69. BEFORE ROBOTS RUNTHEM,
ANYONE INTHE COMPANY
SHOULD BE ABLETO

70. PEOPLE SHOULD MAKE HARD
DECISIONS, NOT PUSH
BUTTONS

71. GIVE PEOPLE ACCESS

72. SWE SHOULD HAVE AS
MUCH ACCESS ASTHEY NEED.

73. SWE ALREADY WRITES CODE
THAT HAS ACCESSTO
SENSITIVE DATA

74. PRODUCTION DATA STAYS IN
PRODUCTION

75. MAKE GOOD SYNTHETIC
DATA

76. MAKE GOOD WAYSTOTEST
IN PROD

77. CANARY,A/BTEST, ETC.

78. LEARNTOTRIAGE

79. THINGS BREAK,YOU MUST FIX
THEM

80. MONITORING, METRICS,
OPERATORTOOLS, FAST
BUILD AND DEPLOY

81. TO FIX,YOU NEEDTO KNOW
IT’S BROKEN

82. MONITORING

83. MONITOR APPLICATIONS

84. MONITOR BEHAVIOR

85. STANDARDIZEYOUR METRICS

86. PUSH METRICS OUT

87. DECOUPLEYOUR SYSTEMS

88. WATCH SYSTEMS AS A
FUNCTION OF CAPACITY

89. ONLY ALERT ON SYSTEM
METRICS KNOWNTO HURT
YOU

90. DATA STORES

91. BEWARETHE RDBMS

92. LEARNTO SHARD

93. DITCHTHE DURABILITY
WHEREYOU CAN

94. BUT FIGURE OUT HOWTO
BOOTSTRAP NON-DURABLE
STORES

95. MEMCACHE IS A BLESSING
AND A CURSE

96. ALWAYS CONSIDER A SITE-
WIDE POWER OUTAGE

97. USE DURABLE AND NON-
DURABLE STORESTOGETHER

98. ASKYOUR SRE FOR MORE
INFO

99. DESPITE ALLTHIS,YOU CAN
STILL FAIL...

100. OBVIOUS FAILURE

101. DOWNTIME

102. DOWNTIME WITHOUT
KNOWING

103. NON-OBVIOUS FAILURES

104. HEROIC ACTS

105. WEREYOU UP ALL NIGHT?

106. DIDYOU DOTHAT SAMETASK
ALL DAY?

107. DID A WHOLETEAM STOP
WHATTHEY WERE DOING?

108. THESE ARE HEROIC ACTS,
THEY ARE POISON

109. HEROISM = FAILURE

110. COMES FROM LEGACY
SYSTEMS, PROCEDURES

111. ALSO FROM PERSONALITY
TRAITS...

112. QUESTIONS?
• Grier Johnson
• @grierj
• grierj@gmail.com