For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
8. Docker Challenges
● New workflows for deployments
○ from developer laptop straight to the Cloud
● Orchestration of applications
● Logging
○ Webinars 29/30 September, @sematext
● Monitoring
11. What to monitor?
Cluster
Server 1
Container 1
Container 2
Container N
Container 1
Elasticsearch
Container 2
Node.js - web service
Container N
NGINX - reverse proxy
Server N
JVM Runtime Env
JS Runtime Env
12. The Metrics Stack Changed!
Server Metrics
Runtime Metrics
App Metrics
Server Metrics
Runtime Metrics
App Metrics
Container Metrics
Runtime Metrics = JVM Metrics or Node.js Metrics, …
Without Docker With Docker
13. Poll
What is the result of ‘vmstat’ when started in a
container?
A) OS Metrics limited to the container
B) OS Metrics from the docker host
14. Docker Stats Command
> docker stats $(docker ps -q)
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
19a9f4c9ecb7 0.00% 1.421 MB/3.944 GB 0.04% 0 B/0 B
7f043a32307c 0.00% 1.413 MB/3.944 GB 0.04% 0 B/0 B
889646bd003f 0.11% 4.235 MB/3.944 GB 0.11% 0 B/0 B
9a29f494bf7d 0.27% 156.8 MB/3.944 GB 3.98% 0 B/0 B
Too simple!
15. Docker API for Metrics
> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn"
| nc -U /var/run/docker.sock
HTTP/1.0 200 OK
Date: Thu, 30 Jul 2015 08:37:31 GMT
Content-Type: text/plain; charset=utf-8
{"read":"2015-07-30T04:37:31.13789208-04:00","network":{"rx_bytes":636450,"rx_packets":1944,"
rx_errors":0,"rx_dropped":0,"tx_bytes":495897,"tx_packets":2415,"tx_errors":0,"tx_dropped":0},"
precpu_stats":{"cpu_usage":{"total_usage":0,"percpu_usage":null,"usage_in_kernelmode":0,"
usage_in_usermode":0},"system_cpu_usage":0,"throttling_data":{"periods":0,"throttled_periods":
0,"throttled_time":0}},"cpu_stats ...
Too verbose!
More than 70 Metrics
are exposed
16. Watching a Metric
> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn"
| nc -U /var/run/docker.sock
| logagent -y
| grep rx_bytes
rx_bytes: 667326
rx_bytes: 667354
rx_bytes: 667413
rx_bytes: 667538
github.com/sematext/logagent-js
Which Metrics are
most relevant?
24. Directly on each Docker Host
● The traditional way!
● Full access to the operating system
● Limitations in ’Container only’ systems
- CoreOS
- RancherOS
- Many PaaS Solutions, e.g., DEIS
25. One Container on each Docker Host
● Yes. The Docker Way!
● Portable to any ‘Docker platform’
● Requirements:
- Access to the Host Metrics + Docker API
- Extended privileges (e.g., for SELinux, GCE,...)
27. SPM for Docker Monitoring
apps.sematext.com/spm-reports/registerApplication.do
hub.docker.com/r/sematext/spm-agent-docker/
github.com/sematext/spm-agent-docker
28. Example 1: Run SPM for Docker
docker run -d --name spm-agent
-v /var/run/docker.sock:/var/run/docker.sock
-e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx
-e HOSTNAME=$HOSTNAME
-e SPM_MONITORING_TAGS="project:dev-test,role:demo"
-e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx
--privileged
sematext/spm-agent-docker
29. Example 2: CoreOS + DEIS
● CoreOS has no package manager
● systemd - init system
● fleet - as distributed init system
● etcd - for central configurations
How to distribute a Monitoring
Agent to all CoreOS servers in a cluster?
31. Example 2: Distribute the Agent
# Configure SPM app token
etcdctl set /sematext.com/myapp/spm/token YOUR_SPM_TOKEN
# Download the fleet unit file for SPM
wget
https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service
-O spm-agent.service
# Start SPM Agent in the whole cluster
fleetctl load spm-agent.service
fleetctl start spm-agent.service
32. Example 3: RANCHER OS
● Minimalistic (~25 MB !)
● No package manager
● All system services are containers
● No cluster manager (by default)
● Use restart policy ‘always’ for services
docker run --restart=always … sematext/spm-agent-docker
42. Setup for node.js Apps
Dockerfile:
FROM nodejs
RUN git clone my/app
RUN npm i
RUN npm i spm-agent-nodejs
CMD node -r spm-agent-nodejs app.js
---
docker build -t myapp .
docker run -e SPM_TOKEN=43A... myapp