SpringOne Platform 2018 Six Simple Steps to Service Level Objectives (SLOs) Marie Cosgrove-Davies, Google

1. Six Simple Steps to SLOs
Marie Cosgrove-Davies
Google Cloud
mariecd@google.com

2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
About me
• Product Manager at Google Cloud
• Part of the Stackdriver group
• Previously:
• Product Manager at Pivotal
• Implementation Eng, Support, Eng, SRE
2

3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
You are here because...
• You have a service or system that you care about
• You want to be confident that your users can do what they need to do
• You don’t want your team to be burned out by unactionable pages
3


4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Six Simple Steps to SLOs

5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 1: Think like a PM
Questions boil down to:
• What does “working” mean for my service?
• How can I measure it?
• How can I alert on it?
5


6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Example service: Cat photos web site
Users: People who admire my cats
Step 2: Identify your users
Who are my users?
6


7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 3: Identify user journeys
• How do users interact with my service?
• What does a successful interaction look like?
• List out workflows for each user, identify critical
user journeys (CUJs)
Example critical user journey:
User can see cat photos when they go to the site.
7


8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Vocab break!
• Service Level Indicator (SLI) - a well-defined measure of ‘good enough’
• Service Level Objective (SLO) - a top-line target for good interactions
• Service Level Agreement (SLA) - a contractually defined performance
threshold below which there are consequences (out of scope for this talk)
8

9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 4: Track user journey success with SLIs
How, precisely, do we measure user journey success?
• Write it out in precise English first
• Have technical discussions before coding starts
Example SLI (in English):
When site is loaded, amount of time it takes to
load a photo (latency), as measured by
instrumentation on the page.
9


10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Side-note: measuring reliability (option 1)
1
0

Simple approach:
● Much harder for distributed
request/response services
=Availability =
good minutes
total minutes
● Intuitive for humans
● Relatively easy to
measure for a continuous
binary metric e.g.
machine uptime
○ Is a server that currently does
not get requests up or down?
○ If 1 or 3 servers are down, is the
service up or down?
fraction of time
when service is working

11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Side-note: measuring reliability (option 2)
11

More complex approach:
● Enables these cases:
=Availability =
good interactions
total interactions
● Handles distributed
request/response
services well
○ Is a server that currently does
not get requests up or down?
○ If 1 or 3 servers are down, is the
service up or down?
fraction of real users for whom
service is working

12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 5: Pick SLOs
Pick a number, iterate
• You’re not committed to your first SLO forever
• Users will tell you if your SLO number needs to be
higher
Example service:
• 90% of pageviews have cat photos load within 1 sec
1
2


13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
1
3

“100% is the wrong
reliability target for
basically everything.”
Benjamin Treynor Sloss,
Vice President of 24x7 Engineering, Google

14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Useful tool: error budgets
1
4

● Product management & Engineering define a reliability target (SLO)
● [100% - SLO] is a “budget of unreliability”
(or the error budget)
● Monitoring measures actual reliability
● Control loop for utilizing budget!

15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Calculating error budgets
15
Availability Level
Allowed Unavailability WIndow
per year per quarter per 28 days
90% 36.5 days 9 days 2.8 days
95% 18.25 days 4.5 days 1.4 days
99% 3.65 days 21.6 hours 6.7 hours
99.5% 1.83 days 10.8 hours 3.4 hours
99.9% 8.76 hours 2.16 hours 40.3 minutes
99.95% 4.38 hours 1.08 hours 20.2 minutes
99.99% 52.6 minutes 12.96 minutes 4.0 minutes
99.999% 5.26 minutes 1.30 minutes 24.2 seconds

16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 6a: Operational Shift - Update monitoring
The ultimate goal: have monitoring that tells you if users are sad
• Now you have:
• SLI metric that tells you if things are working
• SLO threshold that tells you if users are sad
• Error budget that tells you how much more your users will take
...so why are you still paging on CPU consumption?
Example: Configure alerting to fire if more than X% of error budget has been
consumed in the past hour
1
6


17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 6b: Operational Shift - Below SLO
If you’re below your SLO, your users are sad.
Change behavior when below SLO:
• Fewer non-reliability feature releases
• Build reliability features
• Pay down tech debt
Example: Cat photos site is below SLO! Maybe it’s finally time to scale up instances
to improve reliability.
1
7


18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
To recap, six steps:
1. Think like a PM
2. Identify your users
3. Identify user journeys
4. Track user success with SLIs
5. Pick SLOs
6. Update your operational practices
a. Error budgets & monitoring
b. Process for when you’re below SLO
1
8


19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Want to do this easily?

20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stackdriver SLO monitoring
2
0

SLI Type
• Latency (and # milliseconds)
• Availability
Using Stackdriver for API SLOs (EAP)

21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stackdriver SLO monitoring
2
1

Period
• Week
• Month
Using Stackdriver for API SLOs (EAP)

22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stackdriver SLO monitoring
2
2

Goal
99.9% - well-engineered software
($$)
99.99% - well-engineered operations
($$$)
99.999% - well-engineered business
($$$$)
Using Stackdriver for API SLOs (EAP)

23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stackdriver SLO monitoring
2
3

Review
• Confirm your SLO
• Submit
• Start tracking!
Using Stackdriver for API SLOs (EAP)

24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stackdriver SLO monitoring
2
4

SLO graphs
• SLI
• Error budget
• SLO compliance
Using Stackdriver for API SLOs (EAP)

25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Additional resources
• SRE books
• Site Reliability Engineering (free online at google.com/sre)
• The Site Reliability Workbook (worth buying)
• Google CRE blog posts
• Pivotal CloudOps blog post
2
5


26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Thank you!