MySQL Monitoring 101 System Monitoring MySQL Monitoring 101

MySQL Monitoring 101 MySQL Monitoring 101 Simple stuff to save your bacon Ronald Bradford Principal - 42SQL MySQL Users Conference Santa Clara - April 2009 http://ronaldbradford.com Version 1.1 22.Apr.2009

MySQL Monitoring 101 What is Your Situation? How do you know your website is down? http://ronaldbradford.com

MySQL Monitoring 101 Is this you? How do you know your website is down? ❖ The users will let me know ❖ That's somebody else's problem ❖ Our site is never down ❖ Email/SMS/Pager http://ronaldbradford.com

MySQL Monitoring 101 Is this you? You have a performance problem now? ❖ How long has it been happening? ❖ Is it a new problem or a re-occurring problem? ❖ Has it gradually become worse over time? http://ronaldbradford.com

MySQL Monitoring 101 Is this you? When did you last recover from a backup? ❖ We need to think about doing backups ❖ We use MySQL replication ❖ That's somebody else's problem ❖ We verify our backups worked (not recovery) http://ronaldbradford.com

MySQL Monitoring 101 Goal http://ronaldbradford.com

MySQL Monitoring 101 Four Goals to take away Monitoring a MySQL Server(s) ❖ Know what to monitor ❖ Know how you can monitor ❖ Learn practices to diagnose problems ❖ Have a foundation of historical information http://ronaldbradford.com

MySQL Monitoring 101 Overview http://ronaldbradford.com

MySQL Monitoring 101 Monitoring Overview ❖ Hardware ❖ Software ❖ MySQL http://ronaldbradford.com

MySQL Monitoring 101 Hardware http://ronaldbradford.com

MySQL Monitoring 101 Know Your Physical Resources ❖ CPU ❖ Memory ❖ Disk ❖ Network http://ronaldbradford.com

MySQL Monitoring 101 Monitoring Hardware Resources Bare Essentials vmstat ❖ iostat ❖ ps ❖ netstat ❖ ifconfig ❖ top ❖ http://ronaldbradford.com

MySQL Monitoring 101 Monitoring Hardware Resources Very Valuable sar ❖ mpstat ❖ dstat ❖ ethtool ❖ http://ronaldbradford.com

MySQL Monitoring 101 Monitoring Hardware Resources Very Valuable ❖ /proc ❖ /cpuinfo ❖ /meminfo ❖ /loadavg http://ronaldbradford.com

MySQL Monitoring 101 Identifying Resource Bottlenecks ❖ Which physical resource is being stressed? ❖ Why? http://ronaldbradford.com

MySQL Monitoring 101 Identifying Resource Bottlenecks Can you spot Memory swapping, Disk I/O and CPU usage changes? $ vmstat 5 kthr memory page disk faults cpu rbw swap free re mf pi po fr de sr s0 s1 s2 s4 in sy cs us sy id 1 0 0 17114496 2962832 63 211 5180 71 71 0 0 0 0 71 0 4763 4625 3183 72 2 26 1 0 0 17115252 2963184 4 6 439 0 0 0 0 0 0 7 0 3251 4021 2795 79 2 20 1 0 0 17115252 2963144 2 1 0 0 0 0 0 0 0 16 0 3748 4427 3049 89 2 10 1 0 0 17115252 2962912 11 11 1360 0 0 0 0 0 0 47 0 4083 4210 2752 79 2 19 0 0 0 17115248 2962744 4 9 1428 0 0 0 0 0 0 10 0 1072 1015 754 17 1 82 1 0 0 17115248 2962664 4 0 0 0 0 0 0 0 0 42 0 3755 3818 2549 69 2 29 1 0 0 17115240 2962520 2 4 246 0 0 0 0 0 0 3 0 3231 3992 2833 79 1 19 1 0 0 17115228 2962400 3 8 1347 0 0 0 0 0 0 10 0 3706 4339 3063 88 2 10 1 0 0 17115220 2962256 16 11 853 0 0 0 0 0 0 52 0 4275 4201 2672 79 2 19 1 0 0 17115220 2962100 3 5 131 0 0 0 0 0 0 7 0 2742 3275 2385 62 2 36 0 0 0 17115220 2962064 3 0 0 0 0 0 0 0 0 29 0 1887 1247 917 19 1 81 1 0 0 17115216 2961664 7 13 1885 0 0 0 0 0 0 18 0 3597 4674 3270 82 2 17 2 0 0 17114344 2961428 75 281 1953 13 13 0 0 0 0 12 0 3697 4746 3203 87 2 11 1 0 0 17114332 2961396 19 14 211 6 6 0 0 0 0 53 0 4409 4476 2862 77 2 21 1 0 0 17115316 2961544 5 7 426 0 0 0 0 0 0 8 0 2658 3478 2371 64 1 34 0 0 0 17115316 2961500 5 0 0 0 0 0 0 0 0 39 0 2310 1802 1268 26 1 73 1 0 0 17115316 2961428 5 7 823 0 0 0 0 0 0 11 0 3454 4473 3096 82 2 16 1 0 0 17115308 2961224 7 11 920 0 0 0 0 0 0 13 0 3569 4377 3024 86 2 12 1 0 0 17115308 2961016 12 9 870 3 3 0 0 0 0 37 0 4687 5227 3250 82 2 17 http://ronaldbradford.com

MySQL Monitoring 101 Customer Examples ❖ Single CPU Kernel on 4 Processor Server $ uptime 10:32:22 up 196 days, 11:46, 2 users, load average: 3.86, 4.08, 4.26 $ uname -a Linux db1 2.4.20-6 #1 Thu Feb 27 10:06:59 EST 2003 i686 i686 i386 GNU/Linux [root@db1 mysql]# cat /proc/cpuinfo processor :0 vendor_id : GenuineIntel cpu family :6 model : 10 model name : Pentium III (Cascades) stepping :1 cpu MHz : 701.636 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level :2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1399.19 http://ronaldbradford.com

MySQL Monitoring 101 Customer Examples ❖ Single CPU Kernel - Disabled from boot loader $ more /boot/grub/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/sda3 # initrd /initrd-version.img #boot=/dev/sda default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz root (hd0,0) kernel /vmlinuz-2.4.20-6 ro root=LABEL=/ initrd /initrd-2.4.20-6.img [root@db1 mysql]# ls /boot/vmlinu vmlinux-2.4.20-6 vmlinux-2.4.20-6smp vmlinuz vmlinuz-2.4.20-6 vmlinuz-2.4.20-6smp http://ronaldbradford.com

MySQL Monitoring 101 Customer Examples ❖ Excessive Network Load (20GB uncompressed backup) $ ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: d Wake-on: d Link detected: yes http://ronaldbradford.com

MySQL Monitoring 101 Customer Examples ❖ Degraded RAID ❖ Who checks RAID status? ❖ How is it checked? http://ronaldbradford.com

MySQL Monitoring 101 Additional Resources • Identify Bottlenecks • CPU • Memory http://ronaldbradford.com/blog/identifying-resource-bottlenecks-cpu-2009-03-31/ http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/ • Check back for additional articles http://ronaldbradford.com

MySQL Monitoring 101 Software http://ronaldbradford.com

MySQL Monitoring 101 Know Your Running Software ❖ Operating System ❖ Database ❖ Other http://ronaldbradford.com

MySQL Monitoring 101 Operating System Specifics ❖ Kernel Version ❖ 32/64 bit Kernel ❖ 32bit software running on 64bit kernel ❖ Review all running software ❖ Should it be running? http://ronaldbradford.com

MySQL Monitoring 101 MySQL http://ronaldbradford.com

MySQL Monitoring 101 MySQL Log Files 1 ❖ Error Log ❖ Could be OS Log ❖ Slow Query Log http://ronaldbradford.com

MySQL Monitoring 101 MySQL Process ❖ Is mysqld running? ❖ What is the Memory Usage? http://ronaldbradford.com

MySQL Monitoring 101 MySQL Memory Usage 2 ❖ RSS and VSZ $ ps -eopid,fname,rss,vsz,user,command | grep -e quot;RSSquot; -e quot;mysqlquot; PID COMMAND RSS VSZ USER COMMAND 5463 grep 764 5204 ronald grep -e RSS -e mysql 13894 mysqld_s 596 3936 root /bin/sh /usr/bin/mysqld_safe 13933 mysqld 4787812 5127208 mysql /usr/sbin/mysqld --basedir=/usr -- datadir=/vol/mysql/mysqldata --user=mysql --pid-file=/var/run/mysqld/ mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/ mysqld.sock 13934 logger 608 3840 root logger -p daemon.err -t mysqld_safe -i -t mysqld $ ps -eopid,fname,rss,vsz,user,command | grep quot; mysqld quot; | grep -v grep | awk '{print $3,$4}' 4787820 5127208 http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/ http://ronaldbradford.com

MySQL Monitoring 101 MySQL System Variables 3 ❖ Based on my.cnf configuration ❖ Customizable in running server ❖ SHOW GLOBAL VARIABLES http://dev.mysql.com/doc/refman/5.1/en/show-variables.html http://ronaldbradford.com

MySQL Monitoring 101 MySQL Status Variables 4 ❖ Provide valuable clues to the state of your servers ❖ GLOBAL and SESSION scope ❖ SHOW [GLOBAL|SESSION] STATUS http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com

MySQL Monitoring 101 MySQL Connections 5 ❖ SHOW FULL PROCESSLIST http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com

MySQL Monitoring 101 MySQL Storage Engines 6 ❖ SHOW ENGINE INNODB STATUS ❖ Well incorporated in 5.1 Status Variables http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com

MySQL Monitoring 101 Your MySQL Data 7 ❖ Know your data ❖ Disk footprint ❖ Growth http://ronaldbradford.com

MySQL Monitoring 101 Instance Summary select table_schema, sum(data_length+index_length)/1024/1024 as total_mb, sum(data_length)/1024/1024 as data_mb, sum(index_length)/1024/1024 as index_mb, count(*) as tables from information_schema.tables group by table_schema order by 2 desc; http://ronaldbradford.com

MySQL Monitoring 101 Instance Summary +----------------------------+----------------+----------------+---------------+--------+ | table_schema | total_mb | data_mb | index_mb | tables | +----------------------------+----------------+----------------+---------------+--------+ | xxxxxxxxxxxxxxxx_531 | 25301.01871777 | 19678.43473339 | 5622.58398438 | 189 | | xxxxxxxxxxxxxxxx_528 | 18983.51311207 | 14739.53361988 | 4243.97949219 | 96 | | stats | 14803.18925285 | 5469.20292473 | 9333.98632813 | 1082 | | xxxxxxxxxxxxxxxx_4 | 9631.98090553 | 7600.90180397 | 2031.07910156 | 1119 | | xxxxxxxxxxxxxxxx_501 | 9141.53529739 | 7263.91127396 | 1877.62402344 | 72 | | system_logs | 7973.60180283 | 3237.59008408 | 4736.01171875 | 1553 | | stats_cpu | 4544.43988991 | 3317.35981178 | 1227.08007813 | 1243 | | xxxxxxxxxxxxxxxx_555 | 4024.49787521 | 3436.00080490 | 588.49707031 | 293 | | stats_xxxxxxxxx | 3712.71884155 | 2159.92196655 | 1552.79687500 | 1536 | ... http://ronaldbradford.com

MySQL Monitoring 101 Schema Summary select table_name,engine,row_format, table_rows, avg_row_length, (data_length+index_length)/1024/1024 as total_mb, (data_length)/1024/1024 as data_mb, (index_length)/1024/1024 as index_mb from information_schema.tables where table_schema= DATABASE() order by 6 desc; http://ronaldbradford.com

MySQL Monitoring 101 Schema Attributes ❖ Collations ❖ Text/Blob ❖ Storage Engines http://ronaldbradford.com

MySQL Monitoring 101 Schema Attributes select table_schema,engine,table_collation, count(*) as tables from information_schema.tables where table_schema=DATABASE() group by table_schema,engine,table_collation; select table_schema,table_name,column_name,data_type from information_schema.columns where table_schema= DATABASE() and ( data_type LIKE '%TEXT' OR data_type like '%BLOB'); http://ronaldbradford.com

MySQL Monitoring 101 Monitoring http://ronaldbradford.com

MySQL Monitoring 101 System Monitoring ❖ dstat $ cat dbc_os.sh #!/bin/sh DATETIME=`date +%y%m%d.%H%M` LOG_DIR=quot;/opt/monitor/log/quot; INTERVAL=5 COUNT=`expr 3600 / $INTERVAL` LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.txt CSV_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.csv MYSQL_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.mysql.csv /usr/bin/dstat --time --cpu --mem --disk --net --proc --page --swap --load --nocolor --noheaders --output $ {CSV_LOG_FILE} $INTERVAL $COUNT > $LOG_FILE 2>&1 sed -e quot;1,7dquot; -e quot;s/-/,/quot; -e quot;s/ /,/quot; ${CSV_LOG_FILE} > ${MYSQL_LOG_FILE} exit 0 http://ronaldbradford.com

MySQL Monitoring 101 System Monitoring ❖ Example dstat output -----time----- ----total-cpu-usage---- ------memory-usage----- -dsk/total-> date/time |usr sys idl wai hiq siq| used buff cach free| read writ> 19-04 17:00:01| 7 2 91 1 0 0|1151M 475M 1850M 480M| 51k 859k> 19-04 17:00:06| 5 0 95 0 0 0|1168M 475M 1850M 462M|5734B 134k> 19-04 17:00:11| 3 0 96 0 0 0|1175M 475M 1850M 456M| 0 168k> 19-04 17:00:16| 4 0 92 3 0 0|1184M 475M 1850M 447M| 20k 625k> 19-04 17:00:21| 3 0 97 0 0 0|1142M 475M 1850M 488M| 0 307k> 19-04 17:00:26| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 9830B> 19-04 17:00:31| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 2458B> 19-04 17:00:36| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 34k> 19-04 17:00:41| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 4915B> 19-04 17:00:46| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 31k> 19-04 17:00:51| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 42k> 19-04 17:00:56| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 30k> 19-04 17:01:01| 1 0 99 0 0 0|1151M 475M 1850M 480M| 0 58k> 19-04 17:01:06| 0 0 100 0 0 0|1151M 475M 1850M 480M| 0 9011B> 19-04 17:01:11| 8 0 91 0 0 0|1153M 475M 1850M 478M| 0 3277B> 19-04 17:01:16| 2 0 99 0 0 0|1152M 475M 1850M 479M| 0 37k> 19-04 17:01:21| 0 0 100 0 0 0|1152M 475M 1850M 479M| 0 6554B> http://ronaldbradford.com

MySQL Monitoring 101 MySQL Monitoring ❖ hourly.sh ❖ OS ❖ vmstat ❖ ps ❖ uptime ❖ MySQL ❖ Variables ❖ Status ❖ Process List ❖ Innodb Engine Status ❖ Table Status http://ronaldbradford.com

MySQL Monitoring 101 MySQL Analysis ❖ statpack - Human Readable STATUS ❖ Database Activity ❖ Statement Activity ❖ Prepared Statements ❖ Admin Commands ❖ Thread Cache ❖ Table Cache ❖ MyISAM Key Cache ❖ .... http://www.markleith.co.uk/?p=21 http://ronaldbradford.com

MySQL Monitoring 101 statpack - Read/Write Ratio =================================================================================================== Uptime: 17 days 17 hours 22 mins Snapshot Period 1: 59 minute interval =================================================================================================== Variable Delta/Percentage Per Second Total =================================================================================================== Statement Activity =================================================================================================== SELECT: 13,503,876 3,798.56 4,298,170,239 (94.25%) INSERT: 91,101 25.63 25,327,062 (0.56%) UPDATE: 782,004 219.97 220,640,296 (4.84%) DELETE: 9,674 2.72 2,485,643 (0.05%) REPLACE: 0 0.00 4,980 (0.00%) INSERT ... SELECT: 0 0.00 0 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 46,422 13.06 13,700,478 (0.30%) ROLLBACK: 0 0.00 13 (0.00%) http://ronaldbradford.com

MySQL Monitoring 101 statpack - Unnecessary SQL ================================================================================================= Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ================================================================================================= Variable Delta/Percentage Per Second Total ================================================================================================= Statement Activity ================================================================================================= SELECT: 16,042 267.37 8,177,050 (46.03%) INSERT: 5,838 97.30 1,826,616 (10.28%) UPDATE: 1,109 18.48 738,546 (4.16%) DELETE: 2,018 33.63 1,374,983 (7.74%) REPLACE: 0 0.00 0 (0.00%) INSERT ... SELECT: 0 0.00 27 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 5,708 95.13 2,161,232 (12.17%) ROLLBACK: 5,746 95.77 3,485,828 (19.62%) http://ronaldbradford.com

MySQL Monitoring 101 statpack - Ineffective Query Cache =================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval =================================================================================================== Variable Delta/Percentage Per Second Total =================================================================================================== Query Cache =================================================================================================== QCache Hits / SELECT: 11.11% QCache Hit/Qcache Insert: 31.24% Qcache Hits/Invalidations: 0.00% SELECTs: 16,042 267.37 8,177,050 Query Cache Hits: 1,257 20.95 1,022,301 Query Cache Inserts: 4,181 69.68 2,250,062 Queries Not Cached: 11,864 197.73 5,932,162 Cache Low Memory Prunes: 0 0.00 0 Total Cache Blocks: -421 -7.02 12,051 Queries In Cache: -595 -9.92 4,194 Cache Free Blocks: 770 12.83 3,628 http://ronaldbradford.com

MySQL Monitoring 101 Graphing Options ❖ RRDtool ❖ gnuplot ❖ Google Charts API ❖ vmplot.sh ❖ http://www.bigdbahead.com/?p=302 ❖ http://ronaldbradford.com/blog/extending- vmplot-2009-03-31/ http://ronaldbradford.com

MySQL Monitoring 101 vmplot.sh output http://ronaldbradford.com

MySQL Monitoring 101 Always graph your results 70% 60% 50% 43% 40% 30% http://ronaldbradford.com

MySQL Monitoring 101 Products http://ronaldbradford.com

MySQL Monitoring 101 MySQL Enterprise Monitor ❖ Commercial Product - $$$ ❖ Developed by Sun/MySQL http://www.mysql.com/products/enterprise/monitor.html http://ronaldbradford.com

MySQL Monitoring 101 Nagios ❖ Network and Application Monitoring ❖ Open Source http://www.nagios.org/ http://ronaldbradford.com

MySQL Monitoring 101 Cacti ❖ Generic Network Graphing ❖ Uses RRDtool ❖ MySQL specific templates http://www.cacti.net/ http://code.google.com/p/mysql-cacti-templates/ http://ronaldbradford.com

MySQL Monitoring 101 Additional ❖ Hyperic ❖ Ganglia ❖ Munin ❖ Big Brother ❖ MONyog ❖ EM7 ❖ Zenoss http://ronaldbradford.com

MySQL Monitoring 101 External Services ❖ Pingdom ❖ WebPerform http://ronaldbradford.com

MySQL Monitoring 101 What's missing ❖ Some Standards ❖ Active Community ❖ Integration for monitoring all stack products ❖ Application specific metrics ❖ e.g. Number of new customers per hour ❖ e.g. Total Sales for day http://ronaldbradford.com

MySQL Monitoring 101 Professional Help is Available ❖ Two decades IT expertise ❖ 10 years in MySQL ❖ System/Data Architecture ❖ Database Performance and Tuning ❖ High Availability and Scalability ❖ Education and Training http://ronaldbradford.com http://ronaldbradford.com