This document summarizes a presentation about application metrics using Prometheus examples. It discusses key metrics concepts like SLIs, SLOs, and SLAs and how they help understand system performance and drive architectural decisions. It provides examples of collecting metrics like response times and counts in PHP using Prometheus client libraries and rendering metrics for Prometheus server. The presentation emphasizes that metrics without actionability are just numbers and the importance of acting on SLO violations.

1. Application Metrics
with Prometheus examples
Rafael Dohms @rdohms Backend Architect @

2. Application Metrics
with Prometheus examples
Rafael Dohms @rdohms Backend Architect @

3. How do you do
metrics?

4. “The Prometheus
Scientist Method”

5. I hope not.

6. jobs.usabilla.com
Rafael Dohms
Staff Engineer
rdohmsdoh.ms

7. FeedbackFeedback
jobs.usabilla.com
Rafael Dohms
Staff Engineer
rdohmsdoh.ms
We are hiring!
jobs.usabilla.com

8. Let’s talk about metrics.
But let’s do it with a
concrete example.

9. Kafka / DDD / Autonomous Microservices / Monitoring

10. Kafka / DDD / Autonomous Microservices / Monitoring

11. Kafka / DDD / Autonomous Microservices / Monitoring

12. Metrics are insights into
the current state of your
application.

13. Metrics tell you if your
service is healthy.

14. Canary Deploys
OksanaLatysheva

15. Metrics tell you what
is wrong.

16. Metrics tell you what
is right.

17. Metrics tell you what
will soon be wrong.

18. Metrics tell you where
to start looking.

19. Site Reliability Engineering

20. SLIs SLOs
◎
SLAs

21. SLIs
Service Level Indicators
“A quantitative measure of some
aspect of your application”
The response time of a request was 150ms
Source: Site Reliability Engineering - O’Reilly

22. SLOs
◎
Service Level Objectives
“A target value or a range of values
for something measured by an SLI”
Request response times should be below 200ms
Source: Site Reliability Engineering - O’Reilly

23. Help you drive architectural
decisions, like optimisation
SLOs
◎
Response time SLO: 150 ms
95th Percentile of Processing time (PHP time): 5ms
As a result we decided to invest more time in exploring the
problem domain and not optimising our stack.

24. SLAs
Service Level Agreements
“An explicit or implicit contract with
your customer,that includes
consequences of missing their SLOs”
The 99th percentile of requests response times should meet our SLO,or
we will refund users
Source: Site Reliability Engineering - O’Reilly

25. Measuring

26. –Etsy Engineering
“If it moves, we track it.”
https://codeascraft.com/2011/02/15/measure-anything-measure-everything/

27. Metrics
Statistics
What is happening right
now?
How often does this
happen?
Telemetry

28. Telemetry
“the process of recording and transmitting the readings of an instrument”

29. Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

30. Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

31. I really miss Ayrton Senna
Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

32. Statistics
Incoming feedback items
with origin information
Telemetry
response time of public
endpoints

33. “If it moves, we track it.”

34. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”

35. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”
Incoming Data
Peak frequency
CPU
Memory
Disk Space
Bandwith
node
PHP
NginX
Database

36. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”
Incoming Data
Peak frequency
CPU
Memory
Disk Space
Bandwith
node
PHP
NginX
Database
Measure Monitoring
Measure measurements

37. Metrics,Everywhere.

39. SLIs

40. Picking good SLIs

41. SLIs may change
according to who is
looking at the data.

42. Understanding the
nature of your system

43. User-Facing
serving system?
availability,throughput,latency

44. Storage System?
availability,durability,latency

45. Big Data Systems?
throughput,end-to-end latency

46. User-Facing and Big Data Systems

47. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
User-Facing and Big Data Systems

48. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
User-Facing and Big Data Systems
More relevant to
development team

49. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
๏Other Metrics
- node,nginx,php-fpm,java metrics
- server metrics: cpu,memory,disk space
- Size of cluster
- Kafka health
User-Facing and Big Data Systems
More relevant to
development team

50. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
๏Other Metrics
- node,nginx,php-fpm,java metrics
- server metrics: cpu,memory,disk space
- Size of cluster
- Kafka health
User-Facing and Big Data Systems
More relevant to
development team
More relevant to
Infrastructure team

51. Picking Targets

52. Target value
SLI value >= target
Target Range
lower bound <= SLI value <= upper bound

53. Don’t pick a target based
on current performance
What is the business need?
What are users trying to achieve?
How much impact does it have on the user experience?

54. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?

55. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?
“Immediate"
“We sell as
real time”
“500ms,too
much HTML“
“I don’t know”

56. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?
“Immediate"
“We sell as
real time”
“500ms,too
much HTML“
“I don’t know”
What is human perception of
immediate? 100ms
Collection API should respond within 150ms

57. Some, but not too many.
can you settle an argument or priority based on it?

58. Don’t over achieve.
The Chubby example.

59. Adapt. Evolve.
re-define SLO’s as your product evolves.

60. Meeting Expectations.

61. Attach consequences
to your Objectives.

62. The night is dark and
full of loopholes.
take a friend from legal with you.

63. Safety Margins.
like setting the alarm 5 minutes before the meeting.

64. Metrics in Practice.

65. prometheus.io

66. Push Model
scale this!

67. Pull Model
scale this!

68. Prometheus
Telemetry Statistics
Prometheus
StatsD,InfluxDB,etc…
+
Long Term Storage

69. GaugeHistogramCounter Summary
Cumulative
metric the
represents a
single number
that only
increases
Samples and
count of
observations
over time
A counter,that
can go up or
down
Same as a
histogram but
with stream of
quantiles over a
sliding window.

70. jimdo/prometheus_client_php

71. reads from /metrics
reads from local storage
writes to local storage
your code
/metrics

72. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

73. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

74. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
APC / APCu
Redis

75. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
namespace
metric name
help
label names
buckets

76. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
measurement
label values

77. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
namespace
metric name
help
labels

78. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

79. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

80. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

81. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

82. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
# HELP my_app_count_total How many...
# TYPE my_app_count_total counter
my_app_count_total{status="200",url="/url"} 6
# HELP my_app_response_time_ms This measures ....
# TYPE my_app_response_time_ms histogram
my_app_response_time_ms_bucket{status="200",url="/url",le="0"} 0
my_app_response_time_ms_bucket{status="200",url="/url",le="10"} 0
my_app_response_time_ms_bucket{status="200",url="/url",le="50"} 1
my_app_response_time_ms_bucket{status="200",url="/url",le="100"} 1
my_app_response_time_ms_bucket{status="200",url="/url",le="+Inf"} 1
my_app_response_time_ms_count{status="200",url="/url"} 1
my_app_response_time_ms_sum{status="200",url="/url"} 16
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

83. –Also Rafael (today)
“I’ll just try this live demo
again.”
http://localhost:9090/graph http://localhost:8180/metrics
–Rafael (yesterday)
“Demos always fail.”
http://localhost:8180/index
https://github.com/rdohms/talk-app-metrics

84. You can’t act on what
you can’t see.

87. Metrics without
actionability are just
numbers on a screen.

88. Act as soon as an
SLO is threatened .

89. Thank you.
Drop me some
feedback at Usabilla
and make this talk
better.
@rdohms
http://slides.doh.ms
we feedback

90. Thank you.
Drop me some
feedback at Usabilla
and make this talk
better.
@rdohms
http://slides.doh.ms
we feedback