3. Containers Are Great…
• less overhead
• faster deployments
• reproducibility of environments
• cost optimizations
• Isolation
• flexibility
4. …But Monitoring Them Is Not Easy
Containers are:
• Isolated
• Self-Contained
• Simple
• Lightweight
• Disposable
5. …But Monitoring Them Is Not Easy
Containers are:
• Isolated
• Self-Contained
• Simple
• Lightweight
• Disposable
Is it possible to respect these properties and
also have great monitoring?
6. • Containers
• Overview of monitoring/troubleshooting options
• command line
• cAdvisor
• Docker stats
• sysdig
• Examples and demos
In This Talk
7. • Containers
• Overview of monitoring/troubleshooting options
• command line
• cAdvisor
• Docker stats
• sysdig
• Examples and demos
In This Talk
8. In This Talk
• Overview of container monitoring/troubleshooting
options
• command line
• cAdvisor
• Docker stats
• sysdig
• Examples and demos
10. Some Things We Want to Monitor
• Resource usage (CPU/Memory/Disk)
• Network activity
• Application activity
• File I/O activity
• Errors/faults
• Topology
11. Resource Usage (CPU/Memory/Disk)
• ps/top/htop from the host
• Pro: always there
• Cons: Very limited container context (full cgroup string only), only work from
the host
• docker ps / docker top
• cAdvisor
• docker stats
• sysdig
13. cAdvisor Pros and Cons
Pros
• easy to install
• works from a container
• Nice local UI
• API
• Integrations (e.g. InfluxDB, Prometheus, Heapster)
Cons:
• (very) Limited set of metrics
• Containers are black boxes
14. Docker Stats + API
• docker stats mysql
• echo -e "GET /containers/mysql/stats HTTP/1.1rn" |
nc -U /var/run/docker.sock
16. Docker Stats Pros and Cons
Pros
• Part of Docker
• Richer than cAdvisor
• REST API
• Integrations
• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin
• commercial vendors
Cons:
• Docker only
• Containers are black boxes
17. sysdig
• Capture system events, filter them, run useful scripts
• strace + tcpdump + lsof + Lua
• Open Source
18. csysdig
• Capture system events, filter them, run useful scripts
• strace + tcpdump + lsof + htop + iftop + Lua
• Open Source
• Nice curses UI
19. Design Goals
• Production-ready
• Simple
• lightweight
• Rich data
• Native support for containers
• Runs in a container
• Natural workflow
Top
‘f’
Cgroups
Docker
docker ps
docker top mysql aux
Resource usage with sysdig:
sysdig –n1000
./sysdig -pc fd.port=80 and evt.buffer contains GET
./sysdig –c topcontainers_net
Top processes
Show cpu, network and disk I/O
Show internal pid
Show etcd and flannel
Go inside the Ubuntu container
Top containers
Select haproxy
Ports
F5
Connections
F5
tcpdump
docker exec -i -t frontend1 /bin/bash
apt-get install tcpdump
Top ports in the machine
Top users for these ports
All connections in the machine
Traffic by container
Connections in the container
Select wordpress4
Top directories
/etc
F5
Errors
ENOENT
F6
Top files for the whole host
Top directories for the whole host
Top files and top directories for cadvisor
Container errors
Drill down into the Ubuntu container
Show error with sysdig
Select wordpress4
Top directories
/etc
F5
Errors
ENOENT
F6
Top files for the whole host
Top directories for the whole host
Top files and top directories for cadvisor
Container errors
Drill down into the Ubuntu container
Show error with sysdig