Presented by: Jack Neely, 42 Lines, Inc. Presented at All Things Open 2020 Abstract: Prometheus is an awesome tool to consume raw data from your application stacks and extract real data about your users' experience and the state of your fleet. But many teams end up throwing more and more data at Prometheus, or other metrics vendors, and just assume that the tool can make sense of the chaos. Well, garbage in makes for garbage out. We will look at the common methodologies for instrumenting resources and HTTP calls and how these can be applied to produce consistency across the environment. Especially in a micro-services style environment, consistency in instrumentation is key for creating debuggable systems. On this journey we discover that Prometheus has problematic support for percentiles, but percentiles are key for understanding our applications. As application stacks grow, metric cardinality and high error rates in percentiles can blind a whole organization. How can we use Prometheus to build consistent 30 day service level objectives across our applications? We will tackle accuracy in percentile calculations, tactics for handling cardinality, and methods to handle long service level object periods.

1. Finding the Golden Signals with Prometheus
Implementing SLOs, Histograms, and Burn Rates
Jack Neely
jjneely@42lines.net
October 20, 2020
42 Lines, Inc.

2. What Is Prometheus?
prometheus.io
33K Stars, 1.2K Watchers, 5.2K Forks
Cloud Native Time Series Database
1

3. Use Prometheus Effectively
Solve these three simple problems before your business scales:
• Methods of Common Instrumentation
• Service Level Objectives and Arbitrary Percentiles
• Effective Alerting and Burn Rates
3

4. Methods of Common Insturmentation
3

5. Goal of Site Wide Instrumentation
Work with your developers to integrate metrics into the base kit used to build a
new project. Create and encourage a metric naming scheme!
Use the Prometheus Naming Best Practices as a guide. Use standard units of
measurement.
Example PromQL: View a Graph of HTTP/API Hits by Service
> sum by (job) (
rate(http_request_duration_seconds_count[1m])
)
4

6. Use a Method!
The USE Method
• Brendan Gregg, 2012
• Hardware or Resource Focused
• Utilization, Saturation, Errors
The RED Method
• Tom Wilkie, 2015
• Services and APIs
• Rate, Errors, Duration
The 4 Golden Signals
• Google’s Site Reliability Engineering
Book, 2016
• RED + S
• Latency, Traffic, Errors, Saturation
5

7. Methods Create Service Level Indicators
These metrics become a standard set of Service Level Indicators (SLIs). Actionable
Service Level Objectives (SLOs) are constructed from SLIs.
• Latency
• Traffic
• Errors
• Saturation
Performance SLO
SuccessRate
TrafficRate is Availability Ratio for ∆t
Capacity Planing
6

8. An Ideal Workflow: SRE Abstraction Layer
Templating
Code Generation
GitOps
ChatOps
CLI Tools
S3
7

9. Building Service Level Objectives with Prometheus
8

10. Articulating a Service Level Objective
1. Service Level Indicator: Usually from a Method
2. Threshold
• ≤ 2 Seconds
• Boolean True
3. Goal: 95% Availability
4. Time Window: 30 Days
Examples
• The service will have 95% availability over the past 30 days.
• The service will return HTTP API results within 2 seconds or less for 95% of
requests over the trailing 30 days.
10

11. PromQL: Availability Ratio SLO Example
- record: job:http_requests_availability_sli_ratio:rate30d
expr: >
sum by (job) (
rate(http_requests_total{status!~"5.."}[30d])
)
/
sum by (job) (
rate(http_requests_total[30d])
)
- alert: ServiceAvailabilitySLO
expr: >
job:http_requests_availability_sli_ratio:rate30d < 0.95
labels:
severity: page
11

12. PromQL: Latency SLO Example
- rule job:http_request_duration_seconds_sli:rate30d_q95
expr: >
histogram_quantile(0.95, sum by (job, le) (
rate(
http_request_duration_seconds_bucket[30d]
)
))
- alert: ServiceLatencySLO
expr: >
job:http_request_duration_seconds_sli:rate30d_q95 > 2
labels:
severity: page
But wait, what’s the problem with Histograms?
12

13. Prometheus Histograms
12

14. Histograms as a Sketch for Percentile Estimation
Histogram of data$graphite_retrevial_time
Seconds
Frequency
0 1 2 3 4 5
0200040006000
Mean
q(.95)
Standard Deviation
R Calculations on Raw Data
• Gamma Distribution
• µ = 0.2902809
• σ = 0.7330352
• q(.5) = 0.0487695
• q(.95) = 1.452352
• Index of q(.95) = 9, 501
13

15. Histograms as a Sketch for Percentile Estimation
Histogram of data$graphite_retrevial_time
Seconds
Frequency
0 1 2 3 4 5
0200040006000
Prometheus Estimation
• sample = 9, 501
• 8th bucket
• 93rd sample of 368
144 Buckets Stopping at
t = 29
14

16. Histograms as a Sketch for Percentile Estimation
Histogram of data$graphite_retrevial_time
Seconds
Frequency
0 1 2 3 4 5
010002000300040005000
Human Defined Histogram
Metric
• sample = 9, 501
• 5th bucket
• 711th sample of 1187
• Error = 234%
Buckets =
{.05, .1, .5, 1, 5, 10, 50, 100}
15

17. Histograms as a Sketch for Percentile Estimation
Histogram of log10(data$graphite_retrevial_time)
Seconds = 10x
Frequency
−4 −3 −2 −1 0 1 2
0100200300400500600700
Mean
q(.95)
Log-Linear Bucketing
Algorithm
16

18. Working Around Prometheus’s Limitations
Make the SLO a parameter to the application and export it!
http_request_slo_seconds 2
Use these 3 Counters:
http_requests_total{handler="foo",client="iOS"} 42
http_requests_errors_total{handler="foo",client="iOS"} 3
http_requests_over_slo_total{handler="foo",client="iOS"} 1
SLO alert for 95% of events processed under 2 seconds for last 30 days:
1 - rate(http_requests_over_slo_total[30d]) /
rate(http_requests_total[30d])
< 0.95
17

19. Alerting on SLOs
17

20. Alerting Terminology
Reset Time: The time between fixing an incident and the monitoring system
resolving the alert. Did the actions really fix the problem?
Detection Time: The time between the start of an incident and generating an
alert.
Precision: The proportion of alerts for a significant event compared to false
positives. Low traffic conditions can cause a loss of precision.
Recall: The proportion of significant events that resulted in alerts.
18

21. Error Budgets
Definition
ErrorBudget = 1 − SLO
Example
0.05 = 1 − 0.95
Example in Percentages
5% = 100% − 95%
5% of requests in the 30 day time window can respond with durations longer than
2 seconds.
19

22. Calculating Error Burn Rates
Definition
BurnRate =
ErrorRatio∆t
ErrorBudget
Ratio of Error Budget Consumed in ∆t
BudgetConsumedRatio∆t =
t
SLO TimeWindow
× BurnRate
Solve for BurnRate to create an alert that fires if 2% of the Error Budget is
consumed in the last hour.
20

23. PromQL: Burn Rate Example
- rule: job:http_requests_over_slo_ratio:rate1h
expr: >
sum by (job) (
rate(http_requests_over_slo_total[1h])
) /
sum by (job) (
rate(http_requests_total[1h])
)
- alert: BurnRateHigh
expr: >
job:http_requests_over_slo_ratio:rate1h > 14.4 * 0.05
labels:
severity: page
annotations:
summary: 2% of Error Budget consumed in the last hour
21

24. Conclusions
21

25. Takeway Points
Build a Schema and Kit as guard rails against garbage in garbage out patterns.
This produces a rule library.
SLO based measuring has many positive and powerful aspects. As they are based
on percentiles we need to get the math right to base good business decisions on
good data. Prometheus presents us with challenges here.
SLOs are actually reports. More like period progress updates than real time
alerting. Also, alerting should fire before we violate our SLO goals.
22

26. References and Links
Google’s Site Reliability Engineering Book, 2016
Google’s Site Reliability Workbook, 2018
42 Lines Consulting, Jack Neely, jjneely@42lines.net
Monarch: Google’s Planet-Scale In-Memory Time Series Database, August 2020
23

27. Thank You!
devops@42lines.net
Practical Operations Podcast: operations.fm
23