提到 Kubernetes 要如何監測時，標準答案就是 Prometheus 裝下去就對了，但在擁有多個 K8S Cluster 時，組態如何管理，Metric 如何保管，AlertManager 如何設定…等該怎麼做比較好？此分享為 smalltown 透過 Prometheus Operator 整合 Thanos 來監控多個 K8S Cluster 的經驗談，期許大家聽完此分享可以縮短摸索的時間，更輕鬆的使用 Prometheus

1. Prometheus Day 2

2. HELLO!I am Tom Tsai aks smalltown
- MaiCoin/AMIS SRE
- Taipei HashiCorp User
Group Organizer
- AWS User Group Taiwan
Staff
2

3. 3
http://bit.ly/taipei-hug

4. Where is Your
4
?

5. Outline
▫ Prometheus Basic
▫ Kubernetes Operator
▫ Prometheus Operator
▫ Thanos
▫ Internal/External Monitoring
5

6. Prometheus Basic

7. Prometheus Introduction
7
Exporter Prometheus Alertmanager

8. Install Prometheus is Easy
▫ CM Tools: Ansible, Chef, Puppet, SaltStack
▫ Kubernetes Addons
▫ DigitalOcean
▫ ...
8

9. Maintain Prometheus is Bothering (1/2)
9
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management

10. Maintain Prometheus is Bothering (2/2)
10
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

11. Exclude All Problems Today!
11

12. Kubernetes Operator

13. Manage Production Service
13
Stateless Stateful
Setup
Scale Out/Up
Backup
Upgrade
Operation

14. K8S Operator
14

15. What is Kubernetes Operator
▫ An Operator is a Software with Specific Application
Knowledge
▫ Extends the Kubernetes through Custom
Controller/Resource
▫ Enable Users to Create, Configure, and Manage
Applications
15

16. 16
What is Kubernetes Operator
An Operator is a Domain Specific Controller

17. 17
Operation of Stateless Service
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
scale up
desired=3 count=1
kubectl
ReplicaSet
app=web, env=prod
desired=3 count=3
Pod
app=web
env=prod

18. Pod
app=web
env=prod
18
Operation of Stateful Service (Operator)
backup
kubectl
etcd Operator
app=3.0.13
size=3 backup-tool
interval=300s

19. Prometheus Operator

20. Maintain Prometheus is Bothering
20
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management

21. Prometheus Operator Overview
21
K8S Service
(Exporter)
Prometheus
Alertmanager
GrafanaK8S CRD:
ServiceMonitor
PrometheusRule

22. 22
K8S CRD Within Prometheus Operator (1/2)
▫ Prometheus
- Manage Prometheus StatefulSet
- Prometheus Configuration Reloader
- Prometheus Rules Reloader
▫ Alertmanager
- Manage AlertManager StatefulSet
- Configuration Reloader

23. 23
K8S CRD Within Prometheus Operator (2/2)
▫ ServiceMonitor
- 1 ServiceMonitor Mapping to 1 Exporter Service
- All ServiceMonitor Compose Prometheus
Configuration
▫ PrometheusRule
- 1 PrometheusRule Mapping to 1 Rule File

24. 24
How to Monitor A New Service
▫ Precondition
- Prometheus Install/Manage By Operator
▫ Add New Exporter and Related Service
▫ Add New ServiceMonitor/PrometheusRule
▫ Use Helm to Deploy, and The Changes Will be Reloaded
Automatically

25. 25
How to Add A New Grafana Dashboard
▫ Cook Dashboard
▫ Export Dashboard Json File
▫ Use Helm to Deploy to Related K8S ConfigMap
▫ Changes Trigger Grafana Watcher Sidecar to Reload
Automatically

26. Maintain Prometheus is Bothering (1/2)
26
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management
▫ Everything Uses Helm to Deploy and Store in Version
Control

27. Thanos

28. Maintain Prometheus is Bothering
28
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

29. Thanos Main Components
▫ Sidecar
▫ Querier
▫ Store
▫ Compactor
29

30. 30
Global View
Disk
Sidecar Prometheus
Disk
Sidecar Prometheus
Deduplicate
Marge
Querier
...
Thanos Store API
Prometheus HTTP API

31. 31
Unlimited Retention (1/2)
Disk
SidecarPrometheus
Blocks
Blocks
Object Storage
Blocks

32. 32
Unlimited Retention (2/2)
Object Storage
Blocks
Querier
Store
Cache
Blocks
Thanos Store API

33. 33
Compaction & Downsampling
Object Storage
Blocks
Compactor
Disk
Blocks Blocks
Blocks

34. 34
Thanos Overview
Object
Storage
Compactor
Store Querier
Disk
Sidecar Prometheus
Disk
Sidecar Prometheus
...5
1 1
2
3
4

35. Maintain Prometheus is Bothering
35
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

36. Prometheus
Operator Support
Thanos
36

37. 37
Current Status
...

38. Internal/External
Monitoring

39. A Stone's Throw Away
▫ Who Monitor Your Monitor (Prometheus)
▫ Multi-Region Monitoring
▫ Some More Complicated Monitor
▫ ...
39

40. 40
The 80/20 Rule
▫ Pay Some Money to Use External Monitor to Avoid
Blind Spot
▫ E.g. Site 24X7, Monitis...etc

41. 41
THANKS!
Any questions?
You can find me at
▫ facebook.com/smalltown0110

42. 42
http://bit.ly/maiamis