- GreatSQL社區原創內容未經授權不得隨意使用，轉載請聯系小編并注明來源，
- GreatSQL是MySQL的國產分支版本，使用上與MySQL一致，
- 作者： Yejinrong/葉金榮
- 文章來源：GreatSQL社區投稿
- 啟用coredump
- 制造一個coredump場景
- 真實故障場景分析跟蹤
上一篇檔案介紹了如何構建gdb跟蹤除錯環境，本文介紹如何根據錯誤日志資訊，跟蹤定位問題可能的原因，以及如何利用coredump檔案查找問題線索，
1. 啟用coredump
程式運行程序中可能會例外終止或崩潰，OS會把程式掛掉時的記憶體狀態記錄下來，寫入core檔案，這就叫 coredump，通過gdb結合core檔案可以方便地進行除錯，
利用core檔案中保留的例外堆疊檔案，能夠幫助研發同學更快定位問題，因此，如果某些故障斷斷續續會出現，建議階段性開啟coredump功能，
想要開啟coredump，需要先修改OS層的幾個設定：
$ ulimit -c unlimited
$ sysctl -w fs.suid_dumpable=2
$ echo "core.%p.%e.%s" > /proc/sys/kernel/core_pattern
同時，將這些修改持久化到相應檔案中（假定MySQL/GreatSQL服務行程的屬主用戶是 mysql）：
$ echo "mysql - core unlimited" >> /etc/security/limits.conf
$ echo "fs.suid_dumpable=2" >> /etc/sysctl.conf
$ echo "kernel.core_pattern=core.%e.%p.%t" >> /etc/sysctl.conf
$ sysctl -p
接下來，修改 my.cnf 組態檔，增加以下兩行內容：
core_file
innodb_buffer_pool_in_core_file=OFF
然后重啟GreatSQL服務行程，即可生效，查詢確認下：
mysql> show global variables like '%core%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| core_file | ON |
| innodb_buffer_pool_in_core_file | OFF |
+---------------------------------+-------+
這樣設定完成后，需要的話會在 datadir 目錄下生成core檔案，
2. 制造一個coredump場景
我們可以給mysqld行程發送 SIGSEGV(11) 信號，即可模擬出coredump的場景，例如：
$ kill -s SIGSEGV `pidof mysqld`
這時查看GreatSQL錯誤日志檔案，以及core檔案，就會發現有coredump：
$ls -la
...
-rw------- 1 mysql mysql 1081147392 Feb 20 22:36 core.mysqld-debug.2658134.1676903816
...
$ less error.log
...
14:36:56 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 1f4232b893100742b7c519df2fa714648c2d76d9
Server Version: 8.0.25-16-debug Source distribution
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4b04
d26]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(handle_fatal_signal+0x2cb) [0x39a7d22]
/lib64/libpthread.so.0(+0x12c20) [0x7fc3e669ac20]
/lib64/libc.so.6(__poll+0x51) [0x7fc3e45c4a41]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(Mysqld_socket_listener::listen_for_connection_event()+0x57) [0x3995195
]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(Connection_acceptor<Mysqld_socket_listener>::connection_event_loop()+0
x30) [0x355a024]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(mysqld_main(int, char**)+0x27d2) [0x354e4a6]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(main+0x20) [0x32de906]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc3e44f6493]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(_start+0x2e) [0x32de82e]
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/
...
在一線的同學，如果需要向研發尋求支持或報告故障時，可以先參考這篇文章 MySQL報障之coredump收集處理流程，需要采集其他幾個資訊：
- 故障時刻的error log，
- 故障產生的core檔案，
- 如果有general log的話，也采集起來（故障時刻往前約1小時或10萬行日志），
- 導致core發生涉及到的表DDL以及相應的SQL陳述句，有必要的話，可能還要同時提供真實資料（或樣例資料），
3. 真實故障場景分析跟蹤
在GreatSQL 8.0.25-15版本（上一個版本）中，InnoDB并行查詢功能在特定場景下存在bug，會導致crash，相應的日志見下：
mysqld-debug: /opt/greatsql-8.0.25/sql/item.cc:6047: virtual void Item_field::make_field(Send_field*): Assertion `item_name.is_set()' failed.
01:59:20 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 1f4232b893100742b7c519df2fa714648c2d76d9
Server Version: 8.0.25-debug Source distribution
Thread pointer: 0x7fb4a9a0b000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fb4f7aa53b0 thread_stack 0x80000
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4b04d26]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(handle_fatal_signal+0x2cb) [0x39a7d22]
/lib64/libpthread.so.0(+0x12c20) [0x7fb5146cac20]
/lib64/libc.so.6(gsignal+0x10f) [0x7fb51253a37f]
/lib64/libc.so.6(abort+0x127) [0x7fb512524db5]
/lib64/libc.so.6(+0x21c89) [0x7fb512524c89]
/lib64/libc.so.6(+0x2fa76) [0x7fb512532a76] #<--從這里網上，都是錯誤資訊處理邏輯
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Item_field::make_field(Send_field*)+0x9e) [0x3338758] #<--從這里往下，才是真正觸發故障的位置，并記住 "0x3338758" 這個指標
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(THD::send_result_metadata(mem_root_deque<Item*> const&, unsigned int)+0x19d) [0x36977ab]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_result_send::send_result_set_metadata(THD*, mem_root_deque<Item*> const&, unsigned int)+0x2d) [0x35f3ff9]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_expression::ExecuteIteratorQuery(THD*)+0x1f1) [0x38d057b]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_expression::execute(THD*)+0xed) [0x38d0d7d]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Sql_cmd_dml::execute_inner(THD*)+0x1c1) [0x381db25]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Sql_cmd_dml::execute(THD*)+0x5c7) [0x381cfab]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(mysql_execute_command(THD*, bool)+0x565c) [0x37a1a2b]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(dispatch_sql_command(THD*, Parser_state*, bool)+0x769) [0x37a3a1d]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1491) [0x3799819]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(do_command(THD*)+0x51c) [0x3797c48]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug() [0x3991168]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug() [0x52e4b22]
/lib64/libpthread.so.0(+0x817a) [0x7fb5146c017a]
/lib64/libc.so.6(clone+0x43) [0x7fb5125ffdc3]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fb4a9a65028): SELECT ... FROM t1 WHERE ... #<-- 這是觸發bug的SQL陳述句
Connection ID (thread ID): 8
Status: NOT_KILLED
按照上面所說的方法，我們采集了所有相關資訊，并能在測驗環境重現上述故障，
接下來，我們利用gdb來定位分析問題原因：
$ gdb path/bin/mysqld-debug path/core.mysqld-debug.2657287.1657270311
GNU gdb (GDB) Red Hat Enterprise Linux 9.2-4.el8
...
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bin/mysqld-debug...
...
[New LWP 2675795]
[New LWP 2675825]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `./bin/mysqld-debug --defaults-extra-file=./my.cnf'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56 return (INTERNAL_SYSCALL_ERROR_P (val, err)
[Current thread is 1 (Thread 0x7fb4f7aa7700 (LWP 2676055))]
(gdb)
(gdb) b *0x3338758 #<-- 上面記下的指標值，前面加個 "*" 號，在這里打上斷點
Breakpoint 1 at 0x3338758: file /opt/greatsql-8.0.25/sql/item.cc, line 6048. #<-- 指向可能觸發問題的原始碼位置
(gdb)
(gdb) bt #<-- 列印詳細backtrace資訊
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000000004b04f1d in my_write_core (sig=6) at /opt/greatsql-8.0.25/mysys/stacktrace.cc:409
#2 0x00000000039a7f84 in handle_fatal_signal (sig=6) at /opt/greatsql-8.0.25/sql/signal_handler.cc:199
#3 <signal handler called>
#4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5 0x00007fb512524db5 in __GI_abort () at abort.c:79
#6 0x00007fb512524c89 in __assert_fail_base (fmt=0x7fb51268d698 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x57a1835 "item_name.is_set()",
file=0x57a1400 "/opt/greatsql-8.0.25/sql/item.cc", line=6047, function=<optimized out>) at assert.c:92
#7 0x00007fb512532a76 in __GI___assert_fail (assertion=0x57a1835 "item_name.is_set()", file=0x57a1400 "/opt/greatsql-8.0.25/sql/item.cc", line=6047,
function=0x57a3a40 "virtual void Item_field::make_field(Send_field*)") at assert.c:101
#8 0x0000000003338758 in Item_field::make_field (this=0x7fb4a9b5bcf8, tmp_field=0x7fb4f7aa2380) at /opt/greatsql-8.0.25/sql/item.cc:6047
#9 0x00000000036977ab in THD::send_result_metadata (this=0x7fb4a9a0b000, list=..., flags=5) at /opt/greatsql-8.0.25/sql/sql_class.cc:2824
#10 0x00000000035f3ff9 in Query_result_send::send_result_set_metadata (this=0x7fb4a9a0fda0, thd=0x7fb4a9a0b000, list=..., flags=5)
at /opt/greatsql-8.0.25/sql/query_result.cc:76
#11 0x00000000038d057b in Query_expression::ExecuteIteratorQuery (this=0x7fb4a9a65178, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_union.cc:1150
#12 0x00000000038d0d7d in Query_expression::execute (this=0x7fb4a9a65178, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_union.cc:1321
#13 0x000000000381db25 in Sql_cmd_dml::execute_inner (this=0x7fb4a9a0fd68, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_select.cc:814
#14 0x000000000381cfab in Sql_cmd_dml::execute (this=0x7fb4a9a0fd68, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_select.cc:585
#15 0x00000000037a1a2b in mysql_execute_command (thd=0x7fb4a9a0b000, first_level=true) at /opt/greatsql-8.0.25/sql/sql_parse.cc:4684
#16 0x00000000037a3a1d in dispatch_sql_command (thd=0x7fb4a9a0b000, parser_state=0x7fb4f7aa41d0, update_userstat=false)
at /opt/greatsql-8.0.25/sql/sql_parse.cc:5284
#17 0x0000000003799819 in dispatch_command (thd=0x7fb4a9a0b000, com_data=https://www.cnblogs.com/greatsql/archive/2023/04/17/0x7fb4f7aa5370, command=COM_QUERY) at /opt/greatsql-8.0.25/sql/sql_parse.cc:1940
#18 0x0000000003797c48 in do_command (thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_parse.cc:1388
#19 0x0000000003991168 in handle_connection (arg=0x7fb4ba094500) at /opt/greatsql-8.0.25/sql/conn_handler/connection_handler_per_thread.cc:307
#20 0x00000000052e4b22 in pfs_spawn_thread (arg=0x7fb511e44320) at /opt/greatsql-8.0.25/storage/perfschema/pfs.cc:2899
#21 0x00007fb5146c017a in start_thread (arg=) at pthread_create.c:479
#22 0x00007fb5125ffdc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
有了這些資訊，研發同學再去跟蹤定位問題根源就會方便很多，
本文簡單演示了如何利用core檔案去跟蹤定位分析可能導致crash的原因，更多有趣實用的方法還有待進一步挖掘，一起探索新世界吧，
Enjoy GreatSQL ??
關于 GreatSQL
GreatSQL是由萬里資料庫維護的MySQL分支，專注于提升MGR可靠性及性能，支持InnoDB并行查詢特性，是適用于金融級應用的MySQL分支版本，
相關鏈接： GreatSQL社區 Gitee GitHub Bilibili
GreatSQL社區：
社區博客有獎征稿詳情：https://greatsql.cn/thread-100-1-1.html
技術交流群：
微信：掃碼添加
GreatSQL社區助手微信好友，發送驗證資訊加群，
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/550384.html
標籤：其他