Your Attention, Please: Better Observability for Distributed Systems - John Feminella

your attention, please
better observability for distributed systems
by: John Feminella
for: SpringOne Tour 2019
at: Columbus, OH
on: February 28, 2019

JOHN FEMINELLA
Pivotal
http://pivotal.io
http://jxf.me · @jxxf

questions?
great!
tweet them: @jxxf
Slack them: @jxf

typical enterprise metrics experience

it’s bad

our goal today
leave you with a toolset to think about better app metrics design
improve signal and reduce noise
improve developer happiness

latency, traffic, errors, saturation
utilization, saturation, errors
rate, errors, durationRED method
Tom Wilkie
USE method
Brendan Gregg
four golden signals
Google SRE book

latency, traffic, errors, saturation
utilization, saturation, errors
rate, errors, duration

disk I/O compute memory

100 streams/instance
1 sample/sec/stream
24 bytes/sample

2,400 bytes/sec/instance

240,000 bytes/sec

~7.573 TB/year

~7.573 TB/year
~175 $/year

/auditevents
/beans
/caches
/conditions
/configprops
/env
/flyway
/health
/httptrace
/info
/integrationgraph
/heapdump
/jolokia
/logfile
/prometheus
/loggers
/liquibase
/metrics
/mappings
/scheduledtasks
/sessions
/shutdown
/threaddump

http://jxf.me · @jxxfSource: Spring Boot docs, YouTube.
registry.gauge(“tickets.sold”,
Tags.empty(),
ticketSalesBucket,
o -> o.Count());

are we measuring the right things?

SLI
(service level indicator)
SLO
(service level objective)
SLA
(service level agreement)
“ should be true...”
“… proportion of the time...”
“…for at least instances...”
“…or else .”

how do we know if we measured the
right things?

three problems

human brains are biased, so metrics are too

some metrics have unclear correlations to health
7.4 hours of sleep
98.9°F
62 bpm

and we’re not always alerting on the right things

lots of variation in system topologies…

… but little variation in how people measure

is there a better way to think about
business metrics?

humans have quirks
1

http://jxf.me · @jxxfMap of the visual cortex [source]

how many?

let’s count
one, two, three, or four?

let’s count
four, five, six, or seven?

2
4
3
1
5

let’s count
six, seven, eight, or nine?

4
2
6
1 5
7
3 8

3 items!
4 items!
Hm… 7?
subitizing!
about equally fast
enumeration
slower

http://jxf.me · @jxxfChecker shadow illusion [source].

apophenia

0
20
40
60
80
100
120
0 10 20 30 40 50 60 70 80 90 100

statistic dataset 1
54.26
47.83
16.76
26.93
-0.06

0
20
40
60
80
100
120
0 20 40 60 80 100 120

statistic dataset 2
54.26
47.83
16.76
26.93
-0.06

0
20
40
60
80
100
120
0 20 40 60 80 100 120
0
20
40
60
80
100
120
0 20 40 60 80 100

statistic dataset 1 dataset 2
54.26 54.26
47.83 47.83
16.76 16.76
26.93 26.93
-0.06 -0.06

http://jxf.me · @jxxfDatasaurus Dozen [source].

statistic value
memory usage 54.26%
CPU usage 47.83%
disk I/O 16.76 MB/s

observation: we need to measure
the fewest metrics possible
the metrics that measure real work done by the system
the metrics that will reveal the most

is measuring more better?

measuring real systems
3

embrace inherent complexity

a different sort of metric

1 2
4
3

1
demand

demand
How much work is the system being asked to do?

1
demand
2
output

output
How much demand is being satisfied?

1
demand
2
output
3
efficiency

efficiency
How well does the system use resources to
produce output?

1
demand
2
output
3
efficiency
4
capacity

capacity
Does the system have enough resources to do the
work demanded?

1
demand
2
output
4
capacity
3
efficiency

cost: must understand the system

database inside a 1,170-pod Kubernetes cluster
queries
received
queries
processed
within SLA

concert ticketing website with 10,000 requests/minute
requests for
ticket
reservations
tickets sold
without
negative UX

1,000-patient hospital’s EMR system
patient
records
requests
patient
records
returned

testing thresholds
4

http://jxf.me · @jxxfRRDtool [source].

http://jxf.me · @jxxfBaron Schwartz [source].

use the story to understand raw data
5

queries
received
queries
processed
4
capacity
3
efficiency

scenario I: load spike

queries
received
goes up 20%
queries
processed
goes up 20%
4
usage up 20%
3
same
efficiency

scenario II: resource usage up

queries
received
goes up 20%
queries
processed
goes up 20%
4
usage up 60%
3
worse
efficiency

scenario III: rising demand

queries
received
goes up 80%
queries
processed
goes up 20%
4
usage up 20%
3
same
efficiency

other metrics are important too

but optimize for your attention first

is it worth it?
5

anecdotal evidence
from three F100 firms…
mean 98% reduction in false positive rates

anecdotal evidence
mean ~1,500% improvement in correlated outcomes

anecdotal evidence
all three have expanded this approach to new systems

anecdotal evidence
all three investing to apply this to previous systems

Takeaways
put humans first and favor metrics that tell a story
attention is the scarcest computational resource of all
there’s always more data, but never enough attention

Thanks!
questions?
email: jxf@jxf.me
Twitter: @jxxf

Your Attention, Please: Better Observability for Distributed Systems - John Feminella

Recommended

Recommended

More Related Content

Similar to Your Attention, Please: Better Observability for Distributed Systems - John Feminella

Similar to Your Attention, Please: Better Observability for Distributed Systems - John Feminella (20)

More from VMware Tanzu

More from VMware Tanzu (20)

Recently uploaded

Recently uploaded (20)

Your Attention, Please: Better Observability for Distributed Systems - John Feminella