Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Dark Art of Container Monitoring
Similar to The Dark Art of Container Monitoring (20)
Recently uploaded
Recently uploaded (20)
The Dark Art of Container Monitoring
- 1. The Dark Art of Container Monitoring Loris Degioanni
- 2. Me Loris Degioanni sysdig’s creator and CEO Past: WinPcap, Wireshark @ldegio
- 3. Containers Are Great… • less overhead • faster deployments • reproducibility of environments • cost optimizations • Isolation • flexibility
- 4. …But Monitoring Them Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight • Disposable
- 5. …But Monitoring Them Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight • Disposable Is it possible to respect these properties and also have great monitoring?
- 6. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 7. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 8. In This Talk • Overview of container monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos
- 9. In This Talk
- 10. Some Things We Want to Monitor • Resource usage (CPU/Memory/Disk) • Network activity • Application activity • File I/O activity • Errors/faults • Topology
- 11. Resource Usage (CPU/Memory/Disk) • ps/top/htop from the host • Pro: always there • Cons: Very limited container context (full cgroup string only), only work from the host • docker ps / docker top • cAdvisor • docker stats • sysdig
- 12. Installing cAdvisor sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
- 13. cAdvisor Pros and Cons Pros • easy to install • works from a container • Nice local UI • API • Integrations (e.g. InfluxDB, Prometheus, Heapster) Cons: • (very) Limited set of metrics • Containers are black boxes
- 14. Docker Stats + API • docker stats mysql • echo -e "GET /containers/mysql/stats HTTP/1.1rn" | nc -U /var/run/docker.sock
- 15. Docker Stats + API • cpu_stats • network • memory_stats • blkio_stats http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get- container-stats-based-on-resource-usage
- 16. Docker Stats Pros and Cons Pros • Part of Docker • Richer than cAdvisor • REST API • Integrations • docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin • commercial vendors Cons: • Docker only • Containers are black boxes
- 17. sysdig • Capture system events, filter them, run useful scripts • strace + tcpdump + lsof + Lua • Open Source
- 18. csysdig • Capture system events, filter them, run useful scripts • strace + tcpdump + lsof + htop + iftop + Lua • Open Source • Nice curses UI
- 19. Design Goals • Production-ready • Simple • lightweight • Rich data • Native support for containers • Runs in a container • Natural workflow
- 23. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp sysdig Docker (optionally) Save to a trace file foo.scap
- 24. Viewing Network/App Activity •cAdvisor •docker stats •iftop/tcpdump/tshark in the container •sysdig
- 25. Viewing File I/O Activity and Errors •iotop/lsof in the container •sysdig
Editor's Notes
- Top ‘f’ Cgroups Docker docker ps docker top mysql aux
- Resource usage with sysdig: sysdig –n1000 ./sysdig -pc fd.port=80 and evt.buffer contains GET ./sysdig –c topcontainers_net Top processes Show cpu, network and disk I/O Show internal pid Show etcd and flannel Go inside the Ubuntu container Top containers
- Select haproxy Ports F5 Connections F5 tcpdump docker exec -i -t frontend1 /bin/bash apt-get install tcpdump Top ports in the machine Top users for these ports All connections in the machine Traffic by container Connections in the container
- Select wordpress4 Top directories /etc F5 Errors ENOENT F6 Top files for the whole host Top directories for the whole host Top files and top directories for cadvisor Container errors Drill down into the Ubuntu container Show error with sysdig
- Select wordpress4 Top directories /etc F5 Errors ENOENT F6 Top files for the whole host Top directories for the whole host Top files and top directories for cadvisor Container errors Drill down into the Ubuntu container Show error with sysdig