Mydbops 9th Opensource Database Meetup - April 2021 Analyze Corefile and backtraces with GDB for Mysql/MariaDB on Linux

1. Analyze corefile and get backtraces
with GDB for MySQL/MariaDB on Linux
Nilnandan Joshi,
Sr. Support Engineer, MariaDB
nilnandan@gmail.com
www.nilinfobin.com

2. Core Dump/File:
A core dump is a disk file containing an image of the process’s memory at the time of
termination. This image can be used in a debugger to inspect the state of the program at
the time that it terminated. The core dump includes key pieces of program state as
processor registers, memory management details, and other processor and operating
system flags and information.So a core dump is a file that can be very useful to understand
the context of a crash.
GDB:
GDB, the GNU Project debugger, allows you to see what is going on `inside' another
program while it executes -- or what another program was doing at the moment it crashed.
i.e
sudo gdb -p `pidof mysqld`
sudo gdb /path/to/mysqld /path/to/coredump
sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn
mysqld) > mysqld_bt_all_threads.txt
sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn
mysqld) > mysqld_full_bt_all_threads.txt
You can use GDB to study:
• InnoDB locks, table locks, metadata locks etc
• server variables, user variables at session level
• Threads and Statements

3. Backtraces:
A backtrace is a summary of how your program got where it is. It shows one line per
frame, for many frames, starting with the currently executing frame (frame zero), followed
by its caller (frame one), and on up the stack.
In a multi-threaded program, GDB by default shows the backtrace only for the current
thread. To display the backtrace for several or all of the threads, use the command thread
apply. For example, if you type thread apply all backtrace(thread apply all bt) , GDB will
display the backtrace for all the threads; this is handy when you debug a core dump of a
multi-threaded program.
Each line in the backtrace shows the frame number and the function name. The program
counter value is also shown—unless you use set print address off. The backtrace also
shows the source file name and line number, as well as the arguments to the function. The
program counter value is omitted if it is at the beginning of the code for that line number.

4. How to enable Coredump:
In order to enable core dumps, you need to set the core_file system variable either on the
command-line or in a relevant server option group in an option file.
[mariadb]
...
core_file
There are some additional details related to using core files on Linux. On some systems there
is a limit on the sizes of core files that can be dumped. So you have to disable some of the
restrictions. To check the system's current system-wide limit, run
ulimit -c
for the current limit of the mysqld process specifically:
sudo cat /proc/$(pidof mysqld)/limits | grep "core file"
If you need to change the core size limit, the method you use depends on how you
start mysqld. i.e mysqld using mysqld_safe
[mysqld_safe]
…
core_file_size=unlimited

5. - mysqld using systemd
sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF
[Service]
LimitCORE=infinity
EOF
sudo systemctl daemon-reload
If you are using Linux, then it can be helpful to change a few settings to alter where the core
files is written and what file name is used. This is done by setting
the kernel.core_pattern and kernel.core_uses_pid attributes.
By default, core file will be generated in MySQL/MariaDB datadir. If you are using a
production system, you probably want to have the core files in a specific directory, not in
the data directory. (Because core file size could be very large somewhere in GBs) so for that
you can set something like,
sudo mkdir /tmp/corefiles
sudo chmod 777 /tmp/corefiles
sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern
sudo echo 1 > /proc/sys/kernel/core_uses_pid
Since mysqld executes setuid, you may have to set fs.suid_dumpable=2 to allow core
dumps on Linux.
sudo echo 2 > /proc/sys/fs/suid_dumpable

6. In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also
been made into a system variable. Previously it was just an option. It's value can be
checked at runtime by executing the following:
SHOW GLOBAL VARIABLES LIKE 'core_file';
However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core
files on some systems as a way to reduce the size. The following buffers are excluded:
InnoDB buffer pool
InnoDB log buffer
Server recv buffer
Query cache
https://mariadb.com/kb/en/enabling-core-dumps/
https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_core-file
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/

7. Other Requirements:
1. GDB need to be installed if it's not already
yum install gdb
2. debug info packages needs to be install for server and common atleast.
i.e
MariaDB-server-debuginfo-10.4.17_10-1.el8.x86_64
MariaDB-common-debuginfo-10.4.17_10-1.el8.x86_64
3. If you can’t run gdb + core dump on customer’s server then you must have the same
mysqld/mariadb binaries including debug-info packages + all dynamically linked libraries +
core file to run gdb in your local machine. Libraries can be checked by ldd command. i.e
[root@nilcentos7 ~]# ldd /usr/sbin/mysqld
linux-vdso.so.1 => (0x00007ffea8ddd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2149b93000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f214995c000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2149736000)
…
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2148810000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f214860c000)

8. Problem:
190328 23:31:08 [ERROR] mysqld got signal 6 ;
...
Server version: 10.2.23-MariaDB-10.2.23+maria~stretch
...
Thread pointer: 0x7ff4d8001f28
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437]
/lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570]
/lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa]
/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922]
…
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72)
[0x55ca41b4b992]
/usr/sbin/
mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15
enum_duplicatesb+0x1206)[0x55ca41b560f6]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a)
[0x55ca41b70e4a]
/usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f]
...

9. Problem:
2021-02-12 8:39:33 11039259 [ERROR] mysqld: Table '(temporary)' is
marked as crashed and should be repaired
2021-02-12 8:41:09 10757048 [ERROR] mysqld: Table '(temporary)' is
marked as crashed and should be repaired
free(): invalid next size (normal)
210212 8:41:09 [ERROR] mysqld got signal 6 ;
...
Server version: 10.4.17-10-MariaDB-enterprise-log
...
Thread pointer: 0x7f179d1103c8
Attempting backtrace. You can use the following information to
find out
where mysqld died. If you see no messages after this, something
went
terribly wrong...
stack_bottom = 0x7f0a020edbe8 thread_stack 0x49000

10. [root@centos8 nil]# gdb mysqld core.2751610
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
…
Type "apropos word" to search for commands related to "word"...
Reading symbols from mysqld...Reading symbols from /usr/lib/debug/
usr/sbin/mysqld-10.4.17_10-1.el8.x86_64.debug...done.
done.
[New LWP 2338279]
[New LWP 2751610]
…
…
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
…
Core was generated by `/usr/sbin/mysqld'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6
[Current thread is 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]

11. (gdb)
(gdb) thread 1 — this is for the crash
[Switching to thread 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]
#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6
(gdb) backtrace — to get the backtrace
#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6
#1 0x0000563bd293aa84 in memcpy (__len=<optimized out>,
__src=<optimized out>, __dest=<optimized out>) at /usr/include/
bits/string_fortified.h:34
#2 write_block_record (info=info@entry=0x7f0aa8635b18,
old_record=old_record@entry=0x0, record=record@entry=0x7f0aa886ddd0
"347377003ngs",
row=row@entry=0x7f0aa8635bb0,
bitmap_blocks=bitmap_blocks@entry=0x7f0aa8635bb0,
head_block_is_read=<optimized out>, row_pos=0x7f0b74edbb30,
undo_lsn=1,
old_record_checksum=0) at /usr/src/debug/MariaDB-10.4.17-10/
src_0/storage/maria/ma_blockrec.c:2750

12. #3 0x0000563bd293c9f7 in allocate_and_write_block_record
(undo_lsn=1, row=0x7f0aa8635bb0, record=0x7f0aa886ddd0
"347377003ngs", info=0x7f0aa8635b18)
at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_blockrec.c:3571
#4 _ma_write_init_block_record (info=0x7f0aa8635b18,
record=0x7f0aa886ddd0 "347377003ngs")
at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_blockrec.c:3611
#5 0x0000563bd2947a81 in maria_write (info=0x7f0aa8635b18,
record=0x7f0aa886ddd0 "347377003ngs")
at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_write.c:157
#6 0x0000563bd237b631 in handler::ha_write_tmp_row
(this=0x7f0aa8800e00, buf=0x7f0aa886ddd0 "347377003ngs")
at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:6645
#7 0x0000563bd2386134 in schema_table_store_record
(thd=0x7f0aa87efb18, table=0x7f0aa8268550)
at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:3929
#8 0x0000563bd238ad2e in get_schema_column_record(THD*,
TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*,
st_mysql_const_lex_string const*) ()
at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:6113

13. #9 0x0000563bd2391e04 in get_all_tables(THD*, TABLE_LIST*, Item*)
() at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:5066
#10 0x0000563bd23930e4 in get_schema_tables_result(JOIN*,
enum_schema_table_state) () at /usr/src/debug/MariaDB-10.4.17-10/
src_0/sql/sql_show.cc:8919
#11 0x0000563bd2378bb7 in JOIN::exec_inner() () at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4438
#12 0x0000563bd2379113 in JOIN::exec (this=0x7f0aa81f9ee8) at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4264
#13 0x0000563bd2377541 in mysql_select(THD*, TABLE_LIST*, unsigned
int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*,
st_order*, unsigned long long, select_result*, st_select_lex_unit*,
st_select_lex*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/
sql_select.cc:4699
#14 0x0000563bd2377e1e in handle_select (thd=0x7f0aa87efb18,
lex=0x7f0aa87f37d0, result=0x7f0aa81f9ec0,
setup_tables_done_option=0)
at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/
sql_select.cc:410
#15 0x0000563bd230f41c in execute_sqlcom_select(THD*, TABLE_LIST*)
() at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:6429
#16 0x0000563bd23177b3 in mysql_execute_command(THD*) () at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:3926

14. #17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18,
rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310,
is_com_multi=<optimized out>,
is_next_command=<optimized out>) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967
#18 0x0000563bd2320bf1 in dispatch_command(enum_server_command,
THD*, char*, unsigned int, bool, bool) ()
at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:1179
#19 0x0000563bd23221f4 in do_command (thd=0x7f0aa87efb18) at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:1357
#20 0x0000563bd2407973 in do_handle_one_connection
(connect=connect@entry=0x563c11465fe8) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1412
#21 0x0000563bd2407a6d in handle_one_connection
(arg=arg@entry=0x563c11465fe8) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1316
#22 0x0000563bd29f430b in pfs_spawn_thread (arg=0x563c1175fcd8)
at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/perfschema/
pfs.cc:1869
#23 0x00007f190a78c14a in start_thread () from /lib64/
libpthread.so.0
#24 0x00007f19085eff23 in clone () from /lib64/libc.so.6

15. (gdb) frame 17 — this is to know more details in that frame(line)
^^
#17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18,
rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310,
is_com_multi=<optimized out>,
is_next_command=<optimized out>) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967
7967
error= mysql_execute_command(thd);
(gdb) print thd->query_string
$1 = {string = {str = 0x7f0aa81f6520 "SELECT COLUMN_NAME FROM
INFORMATION_SCHEMA.Columns where TABLE_NAME = 'my_workflow' and
TABLE_SCHEMA='news'",
length = 111}, cs = 0x563bd342ad20 <my_charset_latin1>}
(gdb)

16. Some important links for the blog posts
by my friend and colleague Valerii Kravchuk
GDB basics for MySQL DB
https://www.slideshare.net/valeriikravchuk1/gdb-basics-for-my-sql-db-as-percona-live-europe-2019
Checking user threads with GDB
http://mysqlentomologist.blogspot.com/2021/01/checking-user-threads-and-temporary.html
http://mysqlentomologist.blogspot.com/2018/03/checking-user-threads-with-gdb-in-mysql.html
http://mysqlentomologist.blogspot.com/2017/07/how-to-find-processlist-thread-id-in-gdb.html
How to explore InnoDB locks with GDB
http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html
http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html

17. Thank you
Any Questions?