talk given at CERN, February 2019, about the prometheus monitoring solution and how it can be used in CentOS stacks

1. Monitor your CentOS stack with
Prometheus
Julien Pivotto (@roidelapluie)
CERN
February 20st, 2019

2. user{name="roidelapluie"} 1
I like Open Source
I like monitoring
I like automation
... and all of that is my daily job at inuits

3. Traditional sysadmin-style
monitoring
Disk is full
No zombie processes
not 100 user connected
ssh works
swap is empty

4. Questions that come after:
It's up but it is performant?
It's down but for everyone?
Its is degraded but are the users impacted?
Is it even relevant?

5. Metrics Monitoring
e.g. traditionally graphite
Gather fine grained data at frequent interval
Make them useful by labelling them ; store
them
Analyze them to understand what is going on

6. Metrics ARE PART OF
monitoring
Do not maintain a metrics + a "traditional
monitoring" stack
Alert from metrics directly!

7. Monitoring source
Metrics
then Logs
then Spans (opentracing)

8. We are in the cloud era.
Here are some buzzwords for you
cloud, API, openstack, devops, docker, bimodal,
stateless, kubernetes, orchestration, automation,
serverless, docker, humanops, ansible, continuous
deployment, cri-o, jenkins, agile, docker, red hat,
containers, virtualization, provisionning, monitoring,
observability...

9. What is the cloud
Scale
Velocity
Change

10. On Premise looks like the
cloud
Nowadays you have no choice.
Scale
Velocity
Change

11. What are the needs
Automation
Scalability

12. Bye bye
all-in-one tools
tools that don't scale
tools you can not automate

13. We need deserve better tools
Our customers ask us to respond fast, in
seconds
We make hundreds of operations per second
What is your monitoring frequency... 5
minutes?

14. Time to get better tools /
protocols

15. Prometheus
https://prometheus.io/

16. Cloud Native
Easy to configure, deploy, maintain
Designed in multiple services
Container ready
Orchestration ready (dynamic config)
Fuzziness

17. Data Centric
A Metric in Prometheus has metadata:
myql_global_status_handlers_total{handler="tmp_write"} 1122
And lots of function to filter, change, remove...
those metadata while fetching them.
=> OpenMetrics.io

18. Open Source
Apache 2.0
Go
Support for multiple OS
Many "exporters":
https://github.com/prometheus/prometheus/wiki/Default-
port-allocations

19. Simple
1 service = 1 things
Takes care of its db (time based retention
and/or disk space based retention)

20. RPM
https://github.com/openshift/prometheus/blob/
master/golang-github-prometheus-
prometheus.spec (I have not tested it)
https://github.com/lest/prometheus-rpm

21. How does it work?

22. How does it work?

23. How does it work?

24. How does it work?

25. How does it work?

26. Exporters
Exporters expose metrics with an HTTP API
Bindings available for many languages
Exporters do not save data ; they are not
"proxies" and don't "cache" anything

27. Exploring Metrics

28. Exploring Metrics

29. Exploring Metrics

30. Exploring Metrics

31. PromQL
mysql_global_status_commands_total

32. PromQL
mysql_global_status_commands_total{command="select"}

33. PromQL
mysql_global_status_commands_total
{command=~"select|set_options"}

34. PromQL
mysql_global_status_commands_total{command=~"select|se
t_options"}

35. PromQL
deriv(mysql_global_status_connections[5m])

36. PromQL
{__name__=~".+innodb.+cache.*"}
predict_linear(mysql_heartbeat_lag_seconds[5m], 60*2)
sum(rate(mysql_global_status_commands_total{command=~"
(commit|rollback)"}[5m])) without (command)

37. Demo
prometheus
node_exporter + file collector
pushgateway
within openshift

38. Pushing metrics
Short lived batches can send metrics to a
"pushgateway"
Pushgateway is a project maintained by the
prometheus maintainers

39. Alerting
Prometheus has recording rules
Record frequent queries or alerting
Alert base on any PromQL queries.

40. One tool does one job...
Prometheus collects data
Exporters expose data
Grafana graphes data
Alertmanager dispatches alerts

41. Alertmanager
Open Source
Same developers as Prometheus
Second service

42. What is the Alertmanager
doing?
Receives alerts
Group them
Inhibits them
Dispatches them
Deals with HA

43. How to alerts?
Email
Some vendors: Slack, Hipchat, VictorOps,
pagerduty, ...
Generic Webhook -> Plug in anything you want

44. High Availability
2 prometheus servers do exact the same job
They send alerts to Alertmanagers
Alertmanagers are clustered not to send the
same notification twice

46. Grafana
Open Source (Apache 2.0)
Web app
Specialized in visualization
Pluggable
Multiple datasources: prometheus, graphite,
influxdb...
Has an API!

47. History of Grafana
Grafana is a fork of Kibana 3 ; used to be JS-
Driven.
Now fully featured, requires a database, multi-
projects/users support, etc...

48. Grafana and Prometheus
Prometheus shipped its own consoles
Now it recommends Grafana and deprecated
its own consoles

49. Grafana Dashboards

50. Grafana Dashboards

51. Time Picker

52. Configure Prometheus in
Grafana

53. Configure Prometheus in
Grafana

54. Prometheus Dashboard

55. Creating Grafana Dashboards
Takes time
Requires deep knowledge of the tools
Improved over time
Easy to share (json + online library)
Try grafonnet-lib!

56. Conclusion
Lots of data that can be explored in many ways
(subqueries are coming)
Trends and deviations are easy to calculate
Can monitor both business and technical
Very convenient to monitor any kind of stack

57. Conclusion
Open Source
Multiple small pieces
Rich ecosystem
For Cloud and On Premise

58. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact
I will give a
prometheus
workshop
at OSMC.de
2019!