Until recently, applications ran on a handful of carefully curated technologies for years to come, deployments were static and teams siloed. In the microservices age, teams are cross-functional, pick the technologies they consider to work best for them, and are held accountable for what they produce. Application deployments are fine-grained, frequent, scalable and fully automated. What a shift and what a challenge for monitoring such environments! In this presentation, you'll learn how to obtain insights from your OpenShift environment by exploring various open-source monitoring, logging, and tracing technologies, including Elasticsearch, Fluentd, Hawkular, Heapster, Jolokia, Kibana, OpenTracing, Prometheus and ZipKin.

1. CHALLENGES IN A MICROSERVICES AGE:
MONITORING, LOGGING AND TRACING ON OPENSHIFT
Martin Etmajer
Technology Lead @Dynatrace
May 4, 2017

2. WHY A CHALLENGE?

3. Microservice A Microservice B Microservice C

4. Microservice B
Runtime Environment Runtime Environment Runtime Environment
Microservice A Microservice C

5. Microservice B Data
Store
Runtime Environment Runtime Environment Runtime Environment
Microservice A Data
Store
Microservice C Data
Store

6. Microservice A Data
Store
Microservice B Data
Store
Microservice C Data
Store
Runtime Environment Runtime Environment Runtime Environment
3 6
+3+2
5 9

7. Microservice A Microservice B Microservice C
Runtime Environment Runtime Environment Runtime Environment
-3-2
Data
Store
3
Data
Store
6
Data
Store
5 9

8. Microservice A
Load Balancing
Microservice B
Load Balancing
Microservice C
Runtime Environment Runtime Environment Runtime Environment
Load Balancing
Data
Store
Data
Store
Data
Store
3 6

9. Microservice A
Load Balancing
Microservice B
Load Balancing
Microservice C
Runtime Environment Runtime Environment Runtime Environment
Load Balancing
Data
Store
Data
Store
Data
Store
3 6

10. Microservice A
Load Balancing
Microservice B
Load Balancing
Microservice C
Runtime Environment Runtime Environment Runtime Environment
Load Balancing
Data
Store
Data
Store
Data
Store
Messaging System
3 6

11. Microservice A
Load Balancing
Microservice B
Load Balancing
Microservice C
Runtime Environment Runtime Environment Runtime Environment
Load Balancing
Messaging System
API Gateways
Data
Store
Data
Store
Data
Store
3 6

12. Microservice A
Load Balancing
Microservice B
Load Balancing
Microservice C
Runtime Environment Runtime Environment Runtime Environment
Load Balancing
Messaging System
API Gateways
Data
Store
Data
Store
Data
Store
languages
application servers,
container runtimes,…
load balancing
discovery & routing
security
messaging
storage
3 6

13. Red Hat „Hello World“ MSA
Demo

17. MONITORING

18. Monitoring
container health checks with OpenShift Liveness Probes
What?
● a liveness probe periodically checks if a container is able to handle requests
● when a condition has not been met within a given timeout, the probe fails
● if a probe fails, the container is killed and subjected to its restart policy

19. Monitoring
container health checks with OpenShift Liveness Probes
How?
…
1 livenessProbe:
2 httpGet:
3 path: /health
4 port: 8080
5 initialDelaySeconds: 15
6 timeoutSeconds: 1
…
set in spec.containers.livenessProbe of Pod config

20. Monitoring
container health checks with OpenShift Liveness Probes
How?
…
1 livenessProbe:
2 httpGet:
3 path: /health
4 port: 8080
5 initialDelaySeconds: 15
6 timeoutSeconds: 1
…
supports HTTP Get, Container Command and TCP Socket types

21. Monitoring
container health checks with OpenShift Liveness Probes
How?
Demo

22. Monitoring
Where are the fancy dashboards?
Source: d3js.org - Data-Driven Documents

23. Heapster Hawkular Metrics
Monitoring
container, pod and node metrics with Heapster and Hawkular Metrics

25. collects resource usage data, etc.

26. exposes metrics via REST

27. supports various
storage backends

28. Heapster
gathers metrics
runs in a Pod
exposes metrics
influxdb
GCL
elasticsearch
GCM
graphite
kafka

29. e.g. Hawkular

30. Hawkular
A collection of open source monitoring components by Red Hat

31. Hawkular
A collection of open source monitoring components by Red Hat
metrics storage engine

32. Hawkular
A collection of open source monitoring components by Red Hat
distributed tracing and
application performance management

33. Hawkular
A collection of open source monitoring components by Red Hat
alerting

34. Hawkular
A collection of open source monitoring components by Red Hat
feed for Jolokia and Prometheus metrics

35. a metrics storage engine
with a REST interface

36. Monitoring
container, pod and node metrics with Heapster and Hawkular Metrics
! ! ! ! !
gathers metricssends metricsvisualizes metrics
Demo

44. Hawkular OpenShift Agent Jolokia Prometheus
Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)

45. feeds Jolokia/Prometheus metrics data into Hawkular

46. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
! ! ! ! !
is deployed via a DaemonSet

47. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
! ! ! ! !HOSA HOSA HOSA HOSA HOSA
runs

48. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
! ! ! ! !HOSA HOSA HOSA HOSA HOSA
gathers metricssends metrics
applications

49. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
! ! ! ! !HOSA HOSA HOSA HOSA HOSA
gathers metrics
application containers

54. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
How?
1 kind: ConfigMap
2 apiVersion: v1
3 metadata:
4 name: my-hosa-config
5 data:
6 hawkular-openshift-agent: |
7 endpoints:
8 - type: jolokia 15 - type: prometheus
9 protocol: http 16 protocol: http
10 port: 8778 17 port: 8080
11 path: /jolokia/ 18 path: /metrics/
12 metrics 19 metrics:
13 - name: java.lang:type=Memory 20 - name: process_start_time_seconds
14 type: gauge 21 type: gauge
ConfigMap defines agent configuration

55. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
How?
…
1 spec:
2 volumes:
3 - name: hawkular-openshift-agent
4 configMap:
5 name: my-hosa-config
…
a volume named „hawkular-openshift-agent“ tells HOSA to monitor this Pod
Demo

56. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
How?
1 kind: ConfigMap
2 apiVersion: v1
3 metadata:
4 name: my-hosa-config
5 data:
6 hawkular-openshift-agent: |
7 endpoints:
8 - type: jolokia 15 - type: prometheus
9 protocol: http 16 protocol: http
10 port: 8778 17 port: 8080
11 path: /jolokia/ 18 path: /metrics/
12 metrics 19 metrics:
13 - name: java.lang:type=Memory 20 - name: process_start_time_seconds
14 type: gauge 21 type: gauge
Jolokia ?

57. Jolokia
JMX on Capsaicin
access JMX MBeans remotely via REST

58. $ mv $JOLOKIA_HOME/agents/jolokia.war $TOMCAT_HOME/webapps
$ $TOMCAT_HOME/bin/catalina.sh start
Jolokia
JMX on Capsaicin installs a Jolokia WAR agent into Apache Tomcat

59. Monitoring
application level metrics with Hawkular OpenShift Agent (HOSA)
How?
1 kind: ConfigMap
2 apiVersion: v1
3 metadata:
4 name: my-hosa-config
5 data:
6 hawkular-openshift-agent: |
7 endpoints:
8 - type: jolokia 15 - type: prometheus
9 protocol: http 16 protocol: http
10 port: 8778 17 port: 8080
11 path: /jolokia/ 18 path: /metrics/
12 metrics 19 metrics:
13 - name: java.lang:type=Memory 20 - name: process_start_time_seconds
14 type: gauge 21 type: gauge
Prometheus ?

60. Prometheus
A monitoring system for metrics data

61. Prometheus
A monitoring system for metrics data
metrics storage engine

62. Prometheus
A monitoring system for metrics data
various exporters expose metrics from
nodes, database engines and cloud componentry

63. Prometheus
A monitoring system for metrics data
alerting

64. # build.gradle # Application.java
1 dependencies { 1 public void registerMetricServlet() {
2 compile 'io.prometheus.simpleclient.0.0.21' 2 // Expose Promtheus metrics.
3 compile 'io.prometheus.simpleclient_hotspot.0.0.21' 3
context.addServlet(new ServletHolder
(new MetricsServlet()), "/metrics");
4 compile 'io.prometheus.simpleclient_servlet.0.0.21' 4 // Add metrics about CPU, JVM memory etc.
5 } 5 DefaultExports.initialize();
6 }
Prometheus
A monitoring system for metrics data
exposes default JMX metrics: CPU, Memory, Garbage Collection, etc.

65. Where to start?

66. LOGGING

67. how to process all these data?

68. Logging
application events with Elasticsearch, Fluentd and Kibana (EFK)
FluentdElasticsearch Kibana

69. unifies your logging infrastructure; supports almost 200 plugins

70. ! ! ! ! !
is deployed via a DaemonSet
Logging
application events with Elasticsearch, Fluentd and Kibana (EFK)

71. ! ! ! ! !
deploys
Fluentd Fluentd Fluentd Fluentd Fluentd
Logging
application events with Elasticsearch, Fluentd and Kibana (EFK)

72. application events with Elasticsearch, Fluentd and Kibana (EFK)
! ! ! ! !
watches container logs
Fluentd Fluentd Fluentd Fluentd Fluentd
ingests log data
via configuration
Logging
query
ElasticsearchKibana Application
Demo

75. DISTRIBUTED TRACING

77. design goals

78. Google Dapper
trace
(transaction)
initiator
processes
remote calls
Source: Google Dapper Paper
spans
(timed operations)
annotations,
event data

79. ZipKin
A distributed tracing system timescale
spans

80. a standard API for reporting distributed traces
with various client libraries

81. OpenTracing
A vendor-neutral open standard for distributed tracing
The problem?
● instrumentation is challenging: tracing context must propagate within and between processes
● unreasonable to ask all vendor and FOSS software to be intrumented by a single tracing vendor
The solution?
● allow developers to instrument their own code using OpenTracing client libraries
● have the application maintainer choose the tracing technology via configuration change

82. Hawkular APM
Distributed Tracing and Application Performance Management

84. Jaeger
A distributed tracing system
Source: Jaeger Architecture – jaeger.readthedocs.io

85. SOLVES THE CHALLENGE?

87. Dynatrace
All-in-one monitoring, now!
! ! ! ! !
dynatrace/oneagent

88. Dynatrace
All-in-one monitoring, now!
! ! ! ! !
deploys

91. THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews