Spring Boot Actuator 2.0 & Micrometer

1. 1
Spring Boot Actuator 2.0 &
Micrometer
2018-03-27
Toshiaki Maki (@making / tmaki@pivotal.io)

2. Who am I ?
2
Toshiaki Maki (@making) https://blog.ik.am
Sr. Solutions Architect @Pivotal Japan
Spring / Cloud Foundry / Concourse / Kubernetes
" Spring Boot 2": https://note.ik.am (not free!)

3. What' new in Spring Boot 2
X
• Infrastructure Update (Spring 5, Tomcat 8.5, Jetty
9.4, Hibernete 5.3, HikariCP as a default CP)
• Starters for reactive projects
• Reactive web test support
• new OAuth 2 support
• explicit Spring Security configurations
• Actuator refactoring / Micrometer support
• Kotlin support

4. Agenda
3
• Spring Boot Actuator 2
• Micrometer
• Micrometer
• Histograms and percentiles
• Micrometer API

5. Spring Boot Actuator 2

6. Spring Boot Actuator
5
Additional features to help you monitor and manage
your application in production.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

7. Supported Endpoints
6
• auditevents
• beans
• conditions
• configprops
• env
• flyway
• health
• httptrace
https://demo-micrometer.cfapps.io/actuator
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html
• info
• loggers
• liquibase
• metrics
• mappings
• scheduledtasks
• sessions
• shutdown
• threaddump
• heapdump
• jolokia
• logfile
• prometheus
/actuator/*
•renamed
•added

8. Technology Support
7
Spring Boot Actuator 1.x Spring Boot Actuator 2.x
•Spring MVC •Spring MVC
•Spring WebFlux
•Jersey

9. How to enable endpoints
8
Default exposed endpoints are /info and /health.
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.include=info,health,env
All endpoints are not secured by default.

10. Secure endpoints (Spring MVC)
9
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...

11. Secure endpoints (Spring MVC)
9
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()... spring.security.user.name=hoge
spring.security.user.password=foo
spring.security.user.roles=ACTUATOR

12. Secure endpoints (Spring MVC)
10
http.authorizeRequests()
.requestMatchers(EndpointRequest.to("health", "info"))
.permitAll()
.requestMatchers(EndpointRequest.toAnyEndpoint())
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...

13. Secure endpoints (Spring WebFlux)
11
http.authorizeExchange()
.matchers(EndpointRequest.to("health", "info"))
.permitAll()
.matchers(EndpointRequest.toAnyEndpoint())
.hasRole("ACTUATOR")
.anyExchange()
.permitAll()
.and()
.httpBasic()
.and()...

14. Health Check
12
management.endpoint.health.show-details=when_authorized
when authorized

15. Cloud Foundry Support
13 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-cloudfoundry.html

16. Micrometer

17. Micrometer
15
Move to Micrometer
Think SLF4J, but for Metrics
Instrument without vendor lock-in:
• Prometheus
• Netflix Atlas
• Datadog
• InfluxDB
• ...
Multi-dementional metrics
https://micrometer.io/

18. Supported monitoring systems
16
Server polls Client pushes
Prometheus
Atlas, Datadog, Datadog StatsD,
Influx, SignalFx, Telegraf StatsD,
Wavefront, New Relic
Dimensional
Graphite, Ganglia, JMX, Etsy StatsD,
PCF Metrics 1.4
Hierarchical
push
poll

19. Prometheus
17
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

20. /actuator/prometheus
18
system_cpu_count 4.0
system_load_average_1m 1.64
jvm_memory_max_bytes{area="heap",id="PS Eden Space",}
1.05906176E8
jvm_memory_max_bytes{area="heap",id="PS Survivor Space",}
2.1495808E7
jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.00941312E8
http_server_requests_seconds_count{exception="None",method="GET"
,status="200",uri="/hello",} 4469.0
http_server_requests_seconds_sum{exception="None",method="GET",s
tatus="200",uri="/hello",} 297.942920787
http_server_requests_seconds_max{exception="None",method="GET",s
tatus="200",uri="/hello",} 0.401581731
https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-metrics.html#production-ready-
metrics-meter

21. Prometheus
19

22. Grafana
20

23. On Cloud Foundry
21
@Bean
@Profile("cloud")
public MeterRegistryCustomizer meterRegistryCustomizer(
@Value("${vcap.application.name:}") String name,
@Value("${vcap.application.instance_id:}") String id) {
return registry -> registry.config()
.commonTags("cf_app_name", name,
"cf_app_instance_id", id);
}

24. On Cloud Foundry
22
system_cpu_count{cf_app_instance_id="31388366-...",cf_app_name="
demo-micrometer",}
} 4.0
system_load_average_1m{cf_app_instance_id="31388366-...",cf_app_
name="demo-micrometer",} 1.64
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",cf_app_na
me="demo-micrometer",area="heap",id="PS Eden Space",}
1.05906176E8
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",cf_app_na
me="demo-micrometer",area="heap",id="PS Survivor Space",}
2.1495808E7
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",cf_app_na
me="demo-micrometer",area="heap",id="PS Old Gen",} 3.00941312E8

25. Datadog
23
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-datadog</artifactId>
</dependency>
management.metrics.export.datadog.api-key=YOUR-API-KEY

26. Datadog
24

27. PCF Metrics
25
Bind Metrics Forwarder Service
https://github.com/cloudfoundry/java-buildpack-metric-writer Use Metric Writer 2.4.0+ which comes with JBP 4.10+

28. Histograms and percentiles

29. Histogram
27
management.metrics.distribution.percentiles-
histogram.http.server.requests=true

30. Histogram in Prometheus
28
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
The number of requests that took less than 0.447392426 sec.

31. Query percentiles in Prometheus
29
histogram_quantile(0.9,
sum(rate(http_server_requests_seconds_bucket{status="200"
}[5m])) by (app, uri, le))
https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile()

32. SLA (Service Level Agreement)
30
management.metrics.distribution.sla.http.server.requests=
100ms, 200ms, 400ms

33. SLA in Prometheus
31
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.1",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.2",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.4",} 15687.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0

34. Monitor user experience by Apdex
32
Apdex = (Satisfied requests + Tolerating requests / 2)
/ Total number of requests
Satisfied: The response time is less than or equal to T.
Tolerating: The response time is greater than T and less than or equal to 4T.
Frustrated: The response time is greater than 4T.
https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measuring-user-satisfaction

35. For example, T = 1.2 [sec]
33
Level Multiplier Time (T Example = 1.2)
Satisfied T or less <= 1.2 seconds
Tolerating >T, <= 4T Between 1.2 and 4.8
seconds
Frustrated > 4T Greater than 4.8
seconds
https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measuring-user-satisfaction

36. 34
• During a 2minute period a server handles 200 requests
• T = 1.2 [sec]
• 170 of the requests were handled within 1.2 sec
[Satisfied]
• 20 of the requests were handled between 1.2 sec and
4.8 sec [Tolerating]
• The remaining 10 took longer than 4.8 sec. [Frustrated]
• Appdex = (170 + (20 / 2)) / 200 = 0.9
For example, T = 1.2 [sec]

37. Query Appdex (T=100ms) in Prometheus
35
( sum(rate(http_server_requests_seconds_bucket{le="0.1",
status="200"}[5m])) by (app, uri) +
sum(rate(http_server_requests_seconds_bucket{le="0.4",
status="200"}[5m])) by (app, uri) )
https://prometheus.io/docs/practices/histograms/#apdex-score

38. Monitor Appdex in Grafana
36

39. Client-side Percentile
37
management.metrics.distribution.percentiles.http.server.r
equests=0.5, 0.9, 0.95, 0.99, 0.999
http_server_requests_seconds{...,uri="/hello",quantile="0.5",} 0.050855935
http_server_requests_seconds{...,uri="/hello",quantile="0.9",} 0.051380223
http_server_requests_seconds{...,uri="/hello",quantile="0.95",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.99",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.999",} 0.40265318

40. Percentiles for the specific uri in Prometheus
38
max(http_server_requests_seconds{uri="/hello",
status="200", exception="None"}) by (quantile)

41. Micrometer API

42. Meter (Timer, Counter, Gauge, ...)
40
@Component
public class FooService {
final Counter counter;
public FooService(MeterRegistry registry) {
this.counter = Counter.builder("received.messages")
.register(registry);
}
public void handleMessage() {
this.counter.increment();
}
}
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge

43. Meter (Timer, Counter, Gauge, ...)
41
@Component
public class FooService {
final Timer timer;
public FooService(MeterRegistry registry) {
this.timer = Timer.builder("foo").register(registry);
}
public String foo() {
return this.timer.record(() -> {
// ...
});
}
}
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge

44. @Timed
42
@Component
public class FooService {
@Timed("foo")
public String foo() {
// ...
}
}
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(meterRegistry);
} // Not auto-configured in Spring Boot 2.0
https://github.com/micrometer-metrics/micrometer/issues/361

45. MeterFilter
43
@Bean
public MeterFilter meterFilter() {
return MeterFilter.deny(id -> {
String uri = id.getTag("uri");
return id != null && uri.startsWith("/actuator");
}
);
}
Exclude metrics against Actuator endpoints

46. Add MeterBinders
44
@Bean
public HystrixMetricsBinder hystrixMetricsBinder() {
return new HystrixMetricsBinder();
}
@Bean
public HibernateMetrics hibernateMetrics() {
return new HibernateMetrics(...);
}
Check io.micrometer.core.instrument.binder package
Some binders are auto-configured https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-
metrics.html#production-ready-metrics-meter

47. Enjoy Actuator & Micrometer!
45
Thank you for your attention!
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/
production-ready.html
• https://micrometer.io/docs
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/
production-ready-metrics.html
• https://blog.ik.am/entries/448
• https://github.com/making/demo-micrometer