This document provides an overview of optimizing MySQL performance for DevOps. It discusses hardware configuration including memory, disk, CPU and network. It covers important MySQL configuration options like InnoDB settings. It also discusses query tuning techniques like using indexes to improve query performance.
2. • MySQL Support engineer
• Author of
• MySQL Troubleshooting
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova
2
5. • Database server
• 25 years of history
• Popular forks
•
Percona Server for MySQL
•
MariaDB Server
• Replication support from the beginning
What is MySQL?
5
6. Connectors: C, JDBC, ODBC, Python, ...
Connection Pool: Authentication, Caches
SQL interface
Parser
Optimizer
Caches and Buffers:
Global
Engine-specific
Storage engines: InnoDB, MyRocks, ...
File system: Data, Index, logs, other files
•
mysqld and its files
•
Connectors
•
Optimizer
• Caches
• Storage Engines
• Management
MySQL Architecture
6
7. • Bare hardware
• Cloud
• Container-orchestration systems
Where MySQL Installed in 2020?
7
16. • No swapping: sysctl vm.swappiness=1
Memory Configuration
14
17. • No swapping: sysctl vm.swappiness=1
• NUMA interleave: enable in BIOS
Memory Configuration
14
18. • No swapping: sysctl vm.swappiness=1
• NUMA interleave: enable in BIOS
• More is better
Memory Configuration
14
19. • No swapping: sysctl vm.swappiness=1
• NUMA interleave: enable in BIOS
• More is better
•
Memory access is faster than disk access
Memory Configuration
14
20. • No swapping: sysctl vm.swappiness=1
• NUMA interleave: enable in BIOS
• More is better
•
Memory access is faster than disk access
•
Frequently accessed data should be in memory
Memory Configuration
14
21. • Global buffers
• Allocated up to the limit and freed at shutdown
InnoDB Buffer Pool Size
Performance Schema tables
How MySQL Uses Memory
15
22. • Global buffers
• Session buffers
•
Allocated when session starts
• Freed at the disconnect
Connection and result buffers: from net buffer length up to max allowed packet
How MySQL Uses Memory
15
23. • Global buffers
• Session buffers
• Operation-specific buffers
• Allocated for the operation life time
• Can be allocated multiple times
join buffer size: for each tables pair in JOIN
tmp table size: for each temporary table
How MySQL Uses Memory
15
29. • Tables data
•
Log files
• Binary
• Storage engine
InnoDB redo log file
• Error, general, audit, ...
How MySQL Uses Disk
18
30. • Tables data
•
Log files
• Binary
• Storage engine
InnoDB redo log file
• Error, general, audit, ...
•
Disk-based temporary tables
How MySQL Uses Disk
18
31. • Tables data
•
Log files
• Binary
• Storage engine
InnoDB redo log file
• Error, general, audit, ...
•
Disk-based temporary tables
• You may put these on different disks
How MySQL Uses Disk
18
33. •
IO scheduler [noop] or [deadline]
• sudo echo noop >
/sys/block/YOUR DISK/queue/scheduler
or sudo echo deadline >
/sys/block/YOUR DISK/queue/scheduler
CPU Configuration
20
34. •
IO scheduler [noop] or [deadline]
• sudo echo noop >
/sys/block/YOUR DISK/queue/scheduler
or sudo echo deadline >
/sys/block/YOUR DISK/queue/scheduler
•
CPU governor: set to performance
CPU Configuration
20
35. •
IO scheduler [noop] or [deadline]
• sudo echo noop >
/sys/block/YOUR DISK/queue/scheduler
or sudo echo deadline >
/sys/block/YOUR DISK/queue/scheduler
•
CPU governor: set to performance
•
More cores is better
CPU Configuration
20
36. • One thread per connection
•
CPU used only for active threads
How MySQL Uses CPU
21
37. • One thread per connection
•
CPU used only for active threads
• Background work by storage engines
How MySQL Uses CPU
21
40. ? <= CPU cores?
Yes Executed simultaneously
What Happens with Threads
23
41. ? <= CPU cores?
Yes Executed simultaneously
No Wait in a queue
What Happens with Threads
23
42. ? <= CPU cores?
Yes Executed simultaneously
No Wait in a queue
? Does the disk support parallel write?
What Happens with Threads
23
43. ? <= CPU cores?
Yes Executed simultaneously
No Wait in a queue
? Does the disk support parallel write?
Yes Write happens
What Happens with Threads
23
44. ? <= CPU cores?
Yes Executed simultaneously
No Wait in a queue
? Does the disk support parallel write?
Yes Write happens
No Wait in a queue
What Happens with Threads
23
46. • As fast as possible
• Speed of the line
RTT
•
Bandwidth
• Stability
To avoid TCP packet re-submission
Network Configuration
25
47. • As fast as possible
• On the Internet connection
•
Clients can work
• Asynchronous replica will delay
• Synchronous clusters will be not functional
Node disconnects with default options
Very slow response times with adjusted configuration
Network Configuration
25
48. • Communication between server and client
•
Regular client
• Application
• Replication connection (IO) thread
•
Traffic between synchronous nodes
How MySQL Uses Network
26
50. • Server options
•
Components
• Storage engines
InnoDB
•
Plugins
• Server
Binary logging
Optimizer
What can be Configured?
28
51. • Global
• Parameters, necessary for all server processes
Location of server files: datadir etc.
Shared buffers
More
•
Session
• Control connection-specific parameters
MySQL Option Tables
System Variables and Options: Scope
29
53. •
SET [GLOBAL|PERSIST] var = NEW VALUE
•
Command-line option: --var=new value
System Variables: How to Change
30
54. •
SET [GLOBAL|PERSIST] var = NEW VALUE
•
Command-line option: --var=new value
• Configuration file
In the default location
• Specified by option --defaults-file
[mysqld]
var=new_value
System Variables: How to Change
30
55. • PXC Operator Configuration
• In cr.yaml
spec:
secretsName: my-cluster-secrets
pxc:
...
configuration: |
[mysqld]
innodb_log_file_size=8G
System Variables: How to Change
30
56. • PXC Operator Configuration
• In ConfigMap
Create custom configuration file my.cnf
[mysqld]
innodb_log_file_size=8G
System Variables: How to Change
30
57. • PXC Operator Configuration
• In ConfigMap
Create custom configuration file my.cnf
[mysqld]
innodb_log_file_size=8G
Create ConfigMap: kubectl create configmap cluster1-pxc --from-file=my.cnf
System Variables: How to Change
30
58. • PXC Operator Configuration
• In ConfigMap
Create custom configuration file my.cnf
[mysqld]
innodb_log_file_size=8G
Create ConfigMap: kubectl create configmap cluster1-pxc --from-file=my.cnf
Restart PXC
System Variables: How to Change
30
59. • Global options and few session options
< 8.0 A user with privilege SUPER
8.0 + A user with privilege SYSTEM VARIABLES ADMIN
Dynamic Variables: Who Can Change
31
60. • Global options and few session options
< 8.0 A user with privilege SUPER
8.0 + A user with privilege SYSTEM VARIABLES ADMIN
• Session options
< 8.0 Anybody
8.0 + Restricted privileges
SYSTEM VARIABLES ADMIN
SESSION VARIABLES ADMIN
8.0 + Not restricted privileges: anybody
Dynamic Variables: Who Can Change
31
61. • Global options and few session options
< 8.0 A user with privilege SUPER
8.0 + A user with privilege SYSTEM VARIABLES ADMIN
• Session options
< 8.0 Anybody
8.0 + Restricted privileges
SYSTEM VARIABLES ADMIN
SESSION VARIABLES ADMIN
8.0 + Not restricted privileges: anybody
• There are no limits!
Dynamic Variables: Who Can Change
31
62. • Those which control behavior of whole server
• Once at server startup
• Can start with low values, then grow to specified
Buffers: When Allocated
32
63. • Those which control behavior of whole server
• Once at server startup
• Can start with low values, then grow to specified
• Connection options
•
For every connection when connection opens
Buffers: When Allocated
32
64. • Those which control behavior of whole server
• Once at server startup
• Can start with low values, then grow to specified
• Connection options
•
For every connection when connection opens
•
Operation-specific
• For every operation when needed
•
Can be allocated more than once
Buffers: When Allocated
32
67. •
innodb buffer pool size
• innodb log file size
• Should hold changes for an hour
InnoDB
34
68. •
innodb buffer pool size
• innodb log file size
• Should hold changes for an hour
• Too low
InnoDB
34
69. •
innodb buffer pool size
• innodb log file size
• Should hold changes for an hour
• Good
InnoDB
34
70. •
innodb buffer pool size
• innodb log file size
• innodb thread concurrency
• 0 or number of CPU cores
InnoDB
34
71. •
innodb buffer pool size
• innodb log file size
• innodb thread concurrency
• innodb io capacity
•
Default is too small for fast disks
• Up to number of IOPS your disk can handle
InnoDB
34
72. •
innodb buffer pool size
• innodb log file size
• innodb thread concurrency
• innodb io capacity
•
innodb flush method
•
In most cases: O DIRECT
•
Test on your filesystem!
InnoDB
34
74. • Changing these compromize durability!
• innodb flush log at trx commit
1: full ACID, default
Synchronization
35
75. • Changing these compromize durability!
• innodb flush log at trx commit
1: full ACID, default
2: logs written at each commit, flushed per second
MySQL can handle up to 1M INSERTs per second
Safe with PXC and InnoDB Cluster
Synchronization
35
76. • Changing these compromize durability!
• innodb flush log at trx commit
1: full ACID, default
2: logs written at each commit, flushed per second
0: logs are written and flushed once per second
Synchronization
35
77. • Changing these compromize durability!
• innodb flush log at trx commit
1: full ACID, default
2: logs written at each commit, flushed per second
0: logs are written and flushed once per second
• Once per second not guaranteed for 0 and 2
DDL can cause faster flushing
Scheduling may delay flushing
Synchronization
35
78. • Changing these compromize durability!
• innodb flush log at trx commit
• sync binlog
0: Synchronization handled by the system
1: At each transaction commit, default
No transaction lost
N: After N binary log group commits
In case of power or OS crash not flushed transactions can be lost
Synchronization
35
79. •
table open cache
• The number of open tables for all threads
•
Increase when
Connections in the PROCESSLIST are waiting for opening a table
Value of global status variable Opened tables is larger than Open tables
Table Handlers
36
80. •
table open cache
• table definition cache
• Size of the cache for table definitions
• Increase when
Value of Opened table definitions is larger than Open table definitions
Table Handlers
36
81. •
table open cache
• table definition cache
• Increase OS open files limit if needed
Table Handlers
36
83. • You communicate with database using queries
•
Even via NoSQL interface
• They are not SQL queries, but still queries
Heart of the application
38
84. • You communicate with database using queries
•
Even via NoSQL interface
• They are not SQL queries, but still queries
• Data, which you request, matters
•
1,000,000,000 rows
vs 1 row
Heart of the application
38
86. • Selecting a lot of data
• SELECT * FROM many columns table
You May not Be Able to Change Slow Query
40
87. • Selecting a lot of data
• SELECT * FROM many columns table
• Badly written
•
LEFT JOIN instead of INNER JOIN
•
Many values in IN()
• Retrieving large result set, then discarding
•
Not effective SQL
For particular MySQL version
You May not Be Able to Change Slow Query
40
88. • Selecting a lot of data
• SELECT * FROM many columns table
• Badly written
•
LEFT JOIN instead of INNER JOIN
•
Many values in IN()
• Retrieving large result set, then discarding
•
Not effective SQL
For particular MySQL version
•
We still can improve performance
You May not Be Able to Change Slow Query
40
93. • Single column
CREATE INDEX index name ON
the table(the column)
• Multiple columns
CREATE INDEX index name ON
the table(column1, column2)
How to Create an Index
45
94. • Single column
ALTER TABLE table name ADD INDEX
[index name] (the column)
• Multiple columns
ALTER TABLE table name ADD INDEX
[index name] (column1, column2)
How to Create an Index
45
95. •
WHERE the column = a value
• WHERE the column IN(value1, value2,
value3)
• WHERE the column LIKE ’value%’
• WHERE the column LIKE ’%value’
When MySQL Uses Indexes: Conditions
46
96. •
WHERE left part = value1 AND
right part = value2
• WHERE left part = value1 OR
right part = value2
• WHERE right part = value1 AND
left part = value2
•
WHERE right part = value1 OR left part =
value2
When MySQL Uses Indexes: Conditions
46
97. •
table1 JOIN table2 ON table1.column1
= table2.column2
When MySQL Uses Indexes: Joins
47
98. •
table1 JOIN table2 ON table1.column1
= table2.column2
• Same as FROM table1, table2 WHERE
table1.column1 = table2.column2
When MySQL Uses Indexes: Joins
47
99. •
GROUP BY the column
• GROUP BY left part, right part
• GROUP BY right part, left part
• GROUP BY the index, another index
When MySQL Uses Indexes: GROUP BY
48
100. •
ORDER BY the column
• ORDER BY left part, right part
• ORDER BY right part, left part
• ORDER BY the index, another index
When MySQL Uses Indexes: ORDER BY
49
101. 5.7 ORDER BY left part DESC, right part ASC
8.0 ORDER BY left part DESC, right part
ASC
• left part must be descending
• right part must be ascending
• the index(left part DESC, right part ASC)
When MySQL Uses Indexes: ORDER BY
49
102. • Deterministic, built-in
•
Return same value for the same argument
• WHERE the column = FLOOR(123.45)
When MySQL Uses Indexes: Expressions
50
103. • Deterministic, built-in
•
Return same value for the same argument
• WHERE the column = FLOOR(123.45)
• Non-deterministic
•
Return different values for different invocations
• WHERE the column = RAND() ∗ 100
When MySQL Uses Indexes: Expressions
50
104. • Deterministic, built-in
•
Return same value for the same argument
• WHERE the column = FLOOR(123.45)
• Non-deterministic
•
Return different values for different invocations
• WHERE the column = RAND() ∗ 100
•
Stored functions and UDFs
• Indexes are not used
Use generated column indexes
When MySQL Uses Indexes: Expressions
50
105. • Identify which queries do not use indexes
•
Status variables
mysql> select * from performance_schema.session_status
-> where variable_name in (’Created_tmp_tables’,
-> ’Created_tmp_disk_tables’, ’Select_full_join’,
-> ’Select_full_range_join’, ’Select_range’,
-> ’Select_range_check’, ’Select_scan’, ’Sort_merge_passes’,
-> ’Sort_range’, ’Sort_rows’, ’Sort_scan’) and variable_value>0;
+------------------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+------------------------+----------------+
| Select_scan | 2 |
+------------------------+----------------+
Add Indexes
51
106. • Identify which queries do not use indexes
mysql> show global status like ’Handler_read%’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Handler_read_first | 31 |
| Handler_read_key | 1909 |
| Handler_read_last | 0 |
| Handler_read_next | 4393 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 0 |
| Handler_read_rnd_next | 1135 |
+-----------------------+-------+
Add Indexes
51
107. • Identify which queries do not use indexes
•
PERFORMANCE SCHEMA.EVENTS STATEMENTS *
mysql> select * from events_statements_history_long
-> where sql_text like ’select count(*) from employees join %’G
*************************** 1. row ****************************
...
ROWS_SENT: 1 SELECT_RANGE_CHECK: 0
ROWS_EXAMINED: 541058 SELECT_SCAN: 1
CREATED_TMP_DISK_TABLES: 0 SORT_MERGE_PASSES: 0
CREATED_TMP_TABLES: 0 SORT_RANGE: 0
SELECT_FULL_JOIN: 0 SORT_ROWS: 0
SELECT_FULL_RANGE_JOIN: 0 SORT_SCAN: 0
SELECT_RANGE: 0 NO_INDEX_USED: 0
Add Indexes
51
108. • Identify which queries do not use indexes
•
Slow query log
# Time: 2020-10-11T23:34:03.701871Z
# User@Host: root[root] @ localhost [127.0.0.1] Id: 506
# Query_time: 0.024106 Lock_time: 0.000091
Rows_sent: 1 Rows_examined: 1000
SET timestamp=1602459243;
SELECT c FROM sbtest1 WHERE id=996;
Add Indexes
51
109. • Identify which queries do not use indexes
•
QAN in PMM
Add Indexes
51
110. • Identify which queries do not use indexes
• Analyze if adding an index will help
Add Indexes
51
111. • Identify which queries do not use indexes
• Analyze if adding an index will help
• Add the index
Add Indexes
51
112. • Identify which queries do not use indexes
• Analyze if adding an index will help
• Add the index
• Test first!
• ADD INDEX is expensive operation
Use pt-online-schema-change
Add Indexes
51
115. • Since version 8.0
• Collected and used by the Optimizer
Histogram Statistics
53
116. • Since version 8.0
• Collected and used by the Optimizer
• Can be examined in Information Schema
mysql> select HISTOGRAM from information_schema.column_statistics
-> where table_name=’example’G
*************************** 1. row ***************************
HISTOGRAM: {"buckets": [[1, 0.6], [2, 0.8], [3, 1.0]],
"data-type": "int", "null-values": 0.0, "collation-id": 8,
"last-updated": "2018-11-07 09:07:19.791470",
"sampling-rate": 1.0, "histogram-type": "singleton",
"number-of-buckets-specified": 3}
1 row in set (0.00 sec)
Histogram Statistics
53
117. • Since version 8.0
• Collected and used by the Optimizer
• Can be examined in Information Schema
• Affects query execution plan, but not access
• SELECT ... FROM a JOIN b
vs SELECT ... FROM b JOIN a
Histogram Statistics
53
118. • Since version 8.0
• Collected and used by the Optimizer
• Can be examined in Information Schema
• Affects query execution plan, but not access
More details
Histogram Statistics
53
119. 1 2 3 4 5 6 7 8 9 10
0
200
400
600
800
Indexes: Number of Items with Same Value
54
121. 1 2 3 4 5 6 7 8 9 10
0
200
400
600
800
Histograms: Number of Values in Each Bucket
56
122. 1 2 3 4 5 6 7 8 9 10
0
0.2
0.4
0.6
0.8
1
Histograms: Data in the Histogram
57
123. • Data distribution is far from uniform
• Query accesses two or more tables
•
Indexes cannot be used
•
Maintaining them is too expensive
•
They are already used for different condition
Create Histograms
58
126. • Turn ON and OFF particular optimization
Optimizer Switches
60
127. • Turn ON and OFF particular optimization
• Can be not helpful
• Especially for queries, tuned for previous versions
Optimizer Switches
60
128. • Turn ON and OFF particular optimization
• Can be not helpful
• Work with them as with any other option
• Turn OFF and try
SET optimizer_switch = ’use_index_extensions=off’;
SELECT ...
EXPLAIN SELECT ...
Optimizer Switches
60
129. • Turn ON and OFF particular optimization
• Can be not helpful
• Work with them as with any other option
•
If helps implement in queries
SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;
SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;
Optimizer Switches
60
131. • Temporary tables
• Buffers for query execution
•
join buffer size
JOIN conditions, not using indexes
Increase Size of Optimizer Temporary Objects
61
132. • Temporary tables
• Buffers for query execution
•
join buffer size
• read buffer size
Caching indexes for ORDER BY
Bulk insert into partitions
Caching result of nesting queries
Increase Size of Optimizer Temporary Objects
61
140. Troubleshooting hardware resource usage
Troubleshooting configuration issues
MySQL Query Tuning for DevOps
Percona Monitoring and Management
Percona Kubernetes Operators
More information
63