Krzysztof Ksiazek - Severalnines AB
So, you are a developer or sysadmin and showed some abilities in dealing with databases issues. And now, you have been elected to the role of DBA. And as you start managing the databases, you wonder…
* How do I tune them to make best use of the hardware?
* How do I optimize the Operating System?
* How do I best configure MySQL or MariaDB for a specific database workload?
If you're asking yourself the following questions when it comes to optimally running your MySQL or MariaDB databases, then this talk is for you!
We will discuss some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL or MariaDB database. We will also cover some of the variables which are frequently modified even though they should not.
Performance tuning is not easy, especially if you're not an experienced DBA, but you can go a surprisingly long way with a few basic guidelines.
2. Copyright 2018 Severalnines AB
•Tuning process - how to make sure you make correct changes
•Configuration tuning for MariaDB
•InnoDB internals and contentions
Agenda
4. Copyright 2018 Severalnines AB
•Never-ending story which starts once you install MariaDB on the host
•You have to tune for a specific workload
•Workload may change in time
More data can make it I/O-bound
Different query mix may increase CPU load and put stress on different parts of the
InnoDB
•Keep in mind that configuration tuning is not likely to give you a huge increase in
performance (except if the server is really badly configured)
•Make sure you understand why a configuration change ended up with a given result
Tuning process
6. Copyright 2018 Severalnines AB
•You need a deterministic, test environment to make sure you can measure the impact of the
changes
•Environment should mirror production as close as possible, to make it more relevant
•Changes should be introduced one at a time to ensure you understand the impact of each of
them
•Benchmark the system using queries as close to production as possible
•Restore it to the original state for another round of tweaking
•Rinse and repeat until you are happy with results
Tuning process
7. Copyright 2018 Severalnines AB
•Grab a backup of your production systems
•Restore it on a host, restart MariaDB or reboot the host itself to clear caches
•Capture real-world queries using slow log or tcpdump
•Do a baseline run, replay queries using Percona Playback or pt-upgrade
•Restore backup again, restart MariaDB or reboot the host itself to clear caches
•Make _one_ change in my.cnf or OS settings
•Replay queries using Percona Playback or pt-upgrade
•Measure the difference, repeat the process by restoring the backup if you want to make one
more change
Tuning process
9. Copyright 2018 Severalnines AB
Disable Query Cache
Default 10.3 settings
Avg 17987.648 QPS
Default 10.3 settings
Query Cache disabled
Avg 29299.84 QPS
•Query cache - optimize it away by disabling
Use external caching layer (ProxySQL, Redis, Memcached)
All tests were done on r5d.4xlarge
10. Copyright 2018 Severalnines AB
•InnoDB buffer pool - used to cache data
and store dirty pages
•More is better but you need to leave some
memory for other buffers
Per join buffers
Per session buffers
Temporary tables
•You may have heard about 80% rule
It’s more like 90% for large (i.e. 128GB)
hosts
Tuning MariaDB configuration - memory
•Make sure you err on the side of ‘too small’
•Unless you run recent MariaDB (10.2 and
up) where you can resize InnoDB buffer
pool dynamically, without restart
•For fairly loaded (~20-30 running threads)
host with 128GB of memory it should be ok
to leave ~15GB of memory free
•All depends on the workload so your
mileage may vary
11. Copyright 2018 Severalnines AB
•Per-session buffers in InnoDB:
sort_buffer_size, read_buffer_size, read_rnd_buffer_size
•Per-join buffer: join_buffer_size
•By default - small values
•More _not_ always better
At 256KB the way how memory allocates change
smaller chunks use malloc() which is faster than mmap()
•Make sure to benchmark your system after any change to those settings
Tuning MariaDB configuration - memory
16. Copyright 2018 Severalnines AB
•innodb_flush_log_at_trx_commit - governs the durability in InnoDB
1 - full ACID compliance
2 - you may lose up to 1s of transactions when hardware crashes
0 - you may lose up to 1s of transactions when MariaDB crashes
•Significant change in the I/O performance - less flushes means less I/O and less overhead
•Pick whatever you like and whatever you need
Slaves may not require full durability if you have many of them
Galera Cluster nodes may also not require full durability
Tuning MariaDB configuration - I/O performance
17. Copyright 2018 Severalnines AB
•innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number
of disk operations InnoDB can execute
•Set it too low and you may not fully utilize your hardware
•More not always better - aggressive flushing is not always the best option
Redo logs are there for a reason - to minimize number of writes to tablespaces
•innodb_flush_method:
O_DIRECT for BBU-backed hardware
O_DSYNC may work better with SAN
Benchmark your setup before you go live
Tuning MariaDB configuration - I/O performance
23. Copyright 2018 Severalnines AB
•InnoDB Redo Logs are used to store write transactions and they are written sequentially
•MariaDB must not run out of space in them
•Larger logs help with better write merging
•Larger logs help with more stable flushing
•Larger logs may seriously impact recovery time in case of a crash
•The rule of thumb is to make them large enough to store at least 1h of writes
Tuning MariaDB configuration - I/O performance
24. Copyright 2018 Severalnines AB
•max_connections - keep it large enough to handle incoming connections
•If you need to handle thousands of connections, check the connection pooling options or a
proxy, ideally with connection multiplexing (ProxySQL)
•log_bin - you want to have binlogs enabled
Consider sync_binlog=1 Less performance, more durability
•skip_name_resolve - just to make sure your database won’t suffer when DNS will not be
reachable
Tuning MariaDB configuration
28. Copyright 2018 Severalnines AB
•Once you get the output, you can consult source code for your given MariaDB version
Find the lock, understand the context it is located
Decide if there is an option to improve
•In our case, btr0sea.cc:243 points towards the adaptive hash index. Maybe some tuning will
reduce the locking?
InnoDB Internals
29. Copyright 2018 Severalnines AB
•innodb_buffer_pool_instances, table_open_cache_instances
•metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions
•Those options can help you to reduce contention on some of those structures
•Increase number of buffer pools or adaptive hash index partitions if you notice a congestion
on them
•Or, preemptively, if you have to handle highly concurrent traffic
•Don’t use buffer pool instances smaller than 1GB (use 2GB+, too many small instances can
slow down the system)
InnoDB Internals
31. Copyright 2018 Severalnines AB
•Make sure you approach the tuning with a correct process
It requires patience
You should understand the results before making another change
•Think before you act
What workload I have?
What is the bottleneck that I’m facing?
•Proper trending system is a great help
•Don’t forget about other areas to improve
SQL, index hints, optimizer switches
Summary