6. Install & config prometheus on monitoring server
6
2. Config prometheus.yml file
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: linux_hostname1
target_groups:
- targets: [‘ip_addr:9100']
labels:
alias: hostname1
- job_name: mysql_hostname1
target_groups:
- targets: [‘ip_addr:9104']
labels:
alias: hostname1
/opt/prometheus/prometheus.yml
continue right side…
scrape_configs :
you can add server information.
It can be added automatically using ansible playbook.
7. Install & config alertmanager on monitoring server
7
1. Install alertmanager
Download .tar file
https://prometheus.io/download/
$tar –xvf alertmanager-0.1.1.linux-amd64.tar.gz –C /opt
$mv alertmanager-0.1.1.linux-amd64 alertmanager
2. make a slack Webhook url
You can make a slack webhook url for sending a alert message.
https://slack.com/apps/A0F7XDUAZ-incoming-webhooks
9. 9
4. Config rule file(1/4)
/opt/alertmanager/alert.rules
ALERT LinuxLoad1m
IF node_load1 >15
FOR 1m
ANNOTATIONS {summary="Linux Load is at 10"}
ALERT LinuxMemory
IF ((node_memory_MemTotal - (node_memory_MemFree + node_memory_Buffers +
node_memory_Cached))/node_memory_MemTotal) * 100 >85
FOR 1m
ANNOTATIONS {summary="Linux Memory Usage is at 85%"}
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server
10. 10
4. Config rule file(2/4)
/opt/alertmanager/alert.rules
ALERT LinuxDiskUsage
IF ((node_filesystem_size{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"} -
node_filesystem_avail{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"})
/node_filesystem_size{ fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"}) * 100 > 90
FOR 5m
ANNOTATIONS {summary="Linux Disk Usage over 90%"}
ALERT MySQLSlaveLag
IF mysql_slave_lag_seconds > 300
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "Slave lag is too high.", severity="warning" }
Set a alert rule file
Install & config alertmanager on monitoring server
continue next page…
11. 11
4. Config rule file(3/4)
/opt/alertmanager/alert.rules
ALERT MySQLReplicationSQLThreadStatus
IF mysql_slave_status_slave_sql_running==0
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "SQL thread stop", severity="warning"}
ALERT MySQLReplicationIOThreadStatus
IF mysql_slave_status_slave_io_running==0
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "IO thread stop", severity="warning"}
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server
12. 12
4. Config rule file(4/4)
/opt/alertmanager/alert.rules
ALERT MySQLstatus
IF mysql_up==0
FOR 30s
LABELS { severity = "warning" }
ANNOTATIONS { summary = "Mysql Process Down" }
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server
13. 13
5. Add rule file information on prometheus.yml
/opt/prometheus/prometheus.yml
You can add following information between global and scrape_configs section.
global:
rule_files:
- /opt/alertmanager/alert.rules
scrape_configs:
Set a alert rule file
Install & config alertmanager on monitoring server
14. 14
6. Start alertmanager and prometheus
Alertmanager start
#./alertmanager –config.file=simple.yml
Prometheus start
#./prometheus –config.file prometheus.yml –alertmanager.url=http://10.xxx.xxx.xxx:9093
start alertmanager and prometheus
Install & config alertmanager on monitoring server
15. Install & config alertmanager on monitoring server
15
7. Check status of alertmanager and prometheus
Prometheus status
http://10.xxx.xxx.xxx:9090/status
16. Install & config alertmanager on monitoring server
16
7. Check status of alertmanager and prometheus
Alertmanager status
http://10.xxx.xxx.xxx:9093/#/status
17. Auto install & config exporters using Ansible
17
1. Install Ansible
On iOS
$brew install ansible
On Linux
$yum install ansible
19. 19
/etc/hosts
~/ansible_hosts
#DB server lists
10.xxx.xxx.xxx dbhost01
10.xxx.xxx.xxx dbhost02
10.xxx.xxx.xxx dbhost03
[Group1]
dbhost01
dbhost02
dbhost03
system hosts file
ansible hosts file
You should set following information on profile.
export ANSIBLE_INVENTORY=~/ansible_hosts
Auto install & config exporters using Ansible
20. 20
~/ansible/playbooks/prometheus_exporter.yml
- name: Prometheus Install and Configuration
user : user1
hosts: Group1
sudo: yes
tasks:
- name: make directory
file: path=/opt/prometheus_exporters/ state=directory
- name: Copy tar.gz files
copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/{{ item }}
with_items:
- node_exporter-0.12.0.linux-amd64.tar.gz
- mysqld_exporter-0.8.1.linux-amd64.tar.gz
continue next page…
ansible playbook file
for install and config
prometheus exporter
Auto install & config exporters using Ansible
25. 25
~/ansible/playbooks/files/prometheus/
.my.cnf
[client]
host= {{ansible_default_ipv4.address}}
user=mon_user # db user for monitoring
password=mon_user_passwd # db user password
Auto install & config exporters using Ansible
: mysqld_exporter config file
mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO ‘mon_user'@'127.0.0.1' identified by ‘mon_user_passwd';
mysql> GRANT SELECT ON performance_schema.* TO ‘mon_user'@'127.0.0.1';
You have to add db user for monitoring on db server