crash 是目前廣泛使用的 linux 內核崩潰轉儲檔案的分析工具，掌握 crash 的使用技巧，對于分析定位內核崩潰的問題，有著非常重要的作用，本文首先介紹了 crash 的基本概念和安裝方法，其次詳細介紹了如何使用 crash 工具分析內核崩潰轉儲檔案，包括各種常用除錯命令的使用方法，最后以幾個實際作業中遇到的真實案例向讀者展示了 crash 的強大功能，在這篇文章中，既有詳細的工具使用方法，又有豐富的實際案例分析，相信您讀過以后定會受益匪淺，
什么是 crash
如前文所述，當 linux 系統內核發生崩潰的時候，可以通過 kdump 等方式收集內核崩潰之前的記憶體，生成一個轉儲檔案 vmcore，內核開發者通過分析該 vmcore 檔案就可以診斷出內核崩潰的原因，從而進行作業系統的代碼改進，那么 crash 就是一個被廣泛使用的內核崩潰轉儲檔案分析工具，掌握 crash 的使用技巧，對于定位問題有著十分重要的作用，
使用 crash 的先決條件
由于 crash 用于除錯內核崩潰的轉儲檔案，因此使用 crash 需要依賴如下條件：
-
kernel 映像檔案 vmlinux 在編譯的時候必須指定了 -g 引數，即帶有除錯資訊，
-
需要有一個記憶體崩潰轉儲檔案（例如 vmcore），或者可以通過 /dev/mem 或 /dev/crash 訪問的實時系統記憶體，如果 crash 命令列沒有指定轉儲檔案，則 crash 默認使用實時系統記憶體，這時需要 root 權限，
-
crash 支持的平臺處理器包括：x86, x86_64, ia64, ppc64, arm, s390, s390x ( 也有部分 crash 版本支持 Alpha 和 32-bit PowerPC，但是對于這兩種平臺的支持不保證長期維護 )，
-
crash 支持 2.2.5-15（含）以后的 Linux 內核版本，隨著 Linux 內核的更新，crash 也在不斷升級以適應新的內核，
crash 安裝指南
要想使用 crash 除錯內核轉儲檔案，需要安裝 crash 工具和內核除錯資訊包，不同的發行版安裝包名稱略有差異，這里僅列出 RHEL 和 SLES 發行版對應的安裝包名稱如下：
表 1. crash 工具和內核除錯包
| 系統版本 | crash 工具名稱 | 內核除錯資訊包 |
|---|---|---|
| RHEL6.2 | crash | kernel-debuginfo-common kernel-debuginfo |
| SLES11SP2 | crash | kernel-default-debuginfo kernel-ppc64-debuginfo |
以 RHEL 為例，安裝 crash 及內核除錯資訊包的步驟如下：
rpm -ivh crash-5.1.8-1.el6.ppc64.rpm
rpm -ivh kernel-debuginfo-common-ppc64-2.6.32-220.el6.ppc64.rpm
rpm -ivh kernel-debuginfo-2.6.32-220.el6.ppc64.rpm
啟動 crash
啟動引數說明
使用 crash 除錯轉儲檔案，需要在命令列輸入兩個引數：debug kernel 和 dump file，其中 dump file 是內核轉儲檔案的名稱，debug kernel 是由內核除錯資訊包安裝的，不同的發行版名稱略有不同，以 RHEL 和 SLES 為例：
RHEL6.2：/usr/lib/debug/lib/modules/2.6.32-220.el6.ppc64/vmlinux
SLES11SP2：/usr/lib/debug/boot/vmlinux-3.0.13-0.27-ppc64.debug
使用 crash -h 或 man crash 可以查看 crash 支持的一系列選項，這里僅以常用的選項為例說明如下：
-h：列印幫助資訊
-d：設定除錯級別
-S：使用 /boot/System.map 作為默認的映射檔案
-s：不顯示版本、初始除錯資訊等，直接進入命令列
-i file：啟動之后自動運行 file 中的命令，再接受用戶輸入
crash 報告分析
crash 命令啟動后，會產生一個轉儲檔案的分析報告摘要，如下圖所示，
[root@curlylp1 ~]# crash
crash 5.1.8-1.el6
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.32-220.el6.ppc64/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Feb 2 00:31:34 2012
UPTIME: 58 days, 22:52:43
LOAD AVERAGE: 76.11, 77.40, 77.83
TASKS: 481
NODENAME: curlylp1.upt.austin.ibm.com
RELEASE: 2.6.32-220.el6.ppc64
VERSION: #1 SMP Wed Nov 9 08:02:37 EST 2011
MACHINE: ppc64 (5009 Mhz)
MEMORY: 4 GB
PID: 30510
COMMAND: "crash"
TASK: c00000006ddbe460 [THREAD_INFO: c000000073268000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
KERNEL: 系統崩潰時運行的 kernel 檔案
DUMPFILE: 內核轉儲檔案
CPUS: 所在機器的 CPU 數量
DATE: 系統崩潰的時間
TASKS: 系統崩潰時記憶體中的任務數
NODENAME: 崩潰的系統主機名
RELEASE: 和 VERSION: 內核版本號
MACHINE: CPU 架構
MEMORY: 崩潰主機的物理記憶體
PANIC: 崩潰型別，常見的崩潰型別包括：
SysRq (System Request)：通過魔法組合鍵導致的系統崩潰，通常是測驗使用，通過 echo c > /proc/sysrq-trigger，就可以觸發系統崩潰，
oops：可以看成是內核級的 Segmentation Fault，應用程式如果進行了非法記憶體訪問或執行了非法指令，會得到 Segfault 信號，一般行為是 coredump，應用程式也可以自己截獲 Segfault 信號，自行處理，如果內核自己犯了這樣的錯誤，則會彈出 oops 資訊，
crash 內置命令簡介
crash 命令列啟動后，可以通過一些內置命令來列印系統崩潰前的資訊，
bt - backtrace
bt 命令用于查看系統崩潰前的堆疊等資訊，這是系統除錯中非常常用和好用的一個命令，
清單 2. bt 命令結果
crash> bt
PID: 2860 TASK: c0000000677e9550 CPU: 0 COMMAND: "bash"
R0: 0000000000000001 R1: c0000000018978b0 R2: c00000000061c460
R3: c000000001897920 R4: 0000000000000000 R5: 0000000000000000
R6: 0000000000019e07 R7: 0000000000000000 R8: 000000000a000000
R9: c000000072938d80 R10: c0000000006b5d58 R11: c000000000740178
R12: 0000000000000000 R13: c00000000054ea80 R14: 00000000100d0000
R15: 0000000000000000 R16: 00000000100e2ab8 R17: 00000000100b0000
R18: 00000000100d0000 R19: 00000000100d0000 R20: 0000000000000000
R21: 0000000000000000 R22: 00000000100e8a28 R23: 0000000000000000
R24: 8000000000009032 R25: 0000000000000000 R26: 0000000000000000
R27: 0000000000000063 R28: 0000000000000006 R29: 0000000000000000
R30: c00000000058bfe8 R31: c0000000005a5ed0
NIP: c00000000009d9b0 MSR: 8000000000001032 OR3: c000000001897ab0
CTR: c00000000028b6ec LR: c00000000028b708 XER: 0000000000000005
CCR: 0000000000000006 MQ: 0000000000000000 DAR: c0000000005a5ed0
DSISR: c000000001897b10 Syscall Result: 0000000000000000
NIP [c00000000009d9b0] .crash_kexec
LR [c00000000028b708] .sysrq_handle_crashdump
#0 [c0000000018978b0] .crash_kexec at c00000000009d9e0
#1 [c000000001897a90] .sysrq_handle_crashdump at c00000000028b708
#2 [c000000001897b10] .__handle_sysrq at c00000000028b1fc
#3 [c000000001897bc0] .write_sysrq_trigger at c00000000015eadc
#4 [c000000001897c50] .proc_reg_write at c000000000156670
#5 [c000000001897cf0] .vfs_write at c0000000000fd490
#6 [c000000001897d90] .sys_write at c0000000000fdc00
#7 [c000000001897e30] syscall_exit at c0000000000086a4
syscall [c00] exception frame:
R0: 0000000000000004 R1: 00000000ffb6f820 R2: 00000000f7fe95c0
R3: 0000000000000001 R4: 00000000f7d70000 R5: 0000000000000002
R6: 0000000000000001 R7: ffffffffffffffff R8: 0000000000000000
R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000
R12: 0000000000000000 R13: 00000000100dc8e8 R14: 00000000100d0000
R15: 0000000000000000 R16: 00000000100e2ab8 R17: 00000000100b0000
R18: 00000000100d0000 R19: 00000000100d0000 R20: 0000000000000000
R21: 0000000000000000 R22: 00000000100e8a28 R23: 0000000000000000
R24: 0000000000000001 R25: 00000000100e9718 R26: 0000000000000000
R27: 0000000000000002 R28: 000000000ff703f8 R29: 00000000f7d70000
R30: 000000000ff6fff4 R31: 0000000000000002
NIP: 000000000fec0988 MSR: 000000000000d032 OR3: 0000000000000001
CTR: 000000000fe59270 LR: 000000000fe592dc XER: 0000000000000000
CCR: 0000000040242442 MQ: 00000000010b6c30 DAR: 00000000f7d70000
DSISR: 0000000042000000 Syscall Result: 0000000000000000
crash>
如上輸出中，以“# 數字”開頭的行為呼叫堆疊，即系統崩潰前內核依次呼叫的一系列函式，通過這個可以迅速推斷內核在何處崩潰，
log - dump system message buffer
log 命令可以列印系統訊息緩沖區，從而可能找到系統崩潰的線索，log 命令的截圖如下（為節省篇幅，已將部分行省略）：
清單 3. log 命令結果
crash> log
Crash kernel location must be 0x2000000
Using pSeries machine description
Page orders: linear mapping = 24, virtual = 16, io = 12
Found initrd at 0xc000000001500000:0xc000000001c90400
Partition configured for 2 cpus.
Starting Linux PPC64 #1 SMP Tue Jan 24 20:12:50 EST 2012
-----------------------------------------------------
ppc64_pft_size = 0x19
physicalMemorySize = 0x80000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address = 0x0000000000000000
htab_hash_mask = 0x3ffff
-----------------------------------------------------
Linux version 2.6.18-307.el5 ([email protected])
(gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Tue Jan 24 20:12:50 EST 2012
[boot]0012 Setup Arch
Node 0 Memory: 0x0-0x80000000
ps - display process status information
ps 命令用于顯示行程的狀態，（如圖）帶 > 標識代表是活躍的行程，ps 命令的截圖如下（省略部分行）：
清單 4. ps 命令結果
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c00000000054e190 RU 0.0 0 0 [swapper]
0 1 1 c00000007ff15150 RU 0.0 0 0 [swapper]
1 0 1 c00000007ff15960 IN 0.1 4672 2688 init
2 1 0 c00000007ff14940 IN 0.0 0 0 [migration/0]
3 1 0 c00000007ff14130 IN 0.0 0 0 [ksoftirqd/0]
4 1 0 c00000007ff13920 IN 0.0 0 0 [watchdog/0]
5 1 1 c00000007ff13110 IN 0.0 0 0 [migration/1]
6 1 1 c00000007ff12900 IN 0.0 0 0 [ksoftirqd/1]
7 1 1 c00000007ff120f0 IN 0.0 0 0 [watchdog/1]
8 1 0 c00000007ff118e0 IN 0.0 0 0 [events/0]
9 1 1 c00000007ff1ba20 IN 0.0 0 0 [events/1]
10 1 1 c00000007ff110d0 IN 0.0 0 0 [khelper]
139 1 0 c0000000015822f0 IN 0.0 0 0 [kthread]
143 139 0 c000000001c6eb00 IN 0.0 0 0 [kblockd/0]
144 139 1 c000000001580ac0 IN 0.0 0 0 [kblockd/1]
145 139 0 c000000001c6f310 IN 0.0 0 0 [cqueue/0]
146 139 1 c0000000015802b0 IN 0.0 0 0 [cqueue/1]
150 139 0 c00000007ff1e270 IN 0.0 0 0 [khubd]
152 139 0 c00000007ff1ea80 IN 0.0 0 0 [kseriod]
169 1 1 c000000001c62170 IN 0.0 0 0 [rtasd]
209 139 0 c00000007f4ca370 IN 0.0 0 0 [khungtaskd]
> 1771 1 1 c000000001c36a80 RU 0.1 4096 2240 syslogd
dis - disassembling instruction
dis 命令用于對給定地址的內容進行反匯編，dis 命令的截圖如下：
清單 5. dis 命令結果
crash> dis -l c000000000255900
/usr/src/debug/kernel-ppc64-3.0.8/linux-3.0/fs/proc/mmu.c: 47
0xc000000000255900 <.get_vmalloc_info+112>: ld r10,8(r11)
5.5 struct – view data struct
struct 命令用于查看資料結構的定義原型，命令截圖如下：
crash> struct -o vm_struct
struct vm_struct {
[0] struct vm_struct *next;
[8] void *addr;
[16] long unsigned int size;
[24] long unsigned int flags;
[32] struct page **pages;
[40] unsigned int nr_pages;
[48] phys_addr_t phys_addr;
[56] void *caller;
}
SIZE: 64
精彩案例
如前文所述，當 linux 系統內核發生崩潰的時候，可以通過 kdump 等方式收集內核崩潰之前的記憶體，生成一個轉儲檔案 vmcore，內核開發者通過分析該 vmcore 檔案就可以診斷出內核崩潰的原因，從而進行作業系統的代碼改進，那么 crash 就是一個被廣泛使用的內核崩潰轉儲檔案分析工具，掌握 crash 的使用技巧，對于定位問題有著十分重要的作用，
這里采用筆者在實際測驗作業中發現的 SLES 系統下的系統崩潰問題作為案例來進行講解，該系統已經配置了 kdump 啟用，因此在系統發生崩潰之后，在 /var/crash/ 當天日期 / 目錄下面生成一個 vmcore 檔案，下面我們來對這個檔案進行分析，
- 首先啟動 crash
清單 6. 啟動 crash
# crash vmlinux-3.0.8-0.11-ppc64 vmcore
crash 5.1.9
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...
KERNEL: vmlinux-3.0.8-0.11-ppc64
DUMPFILE: vmcore
CPUS: 40
DATE: Wed Nov 16 20:17:11 2011
UPTIME: 10:37:23
LOAD AVERAGE: 60.00, 60.00, 60.00
TASKS: 811
NODENAME: eellp1
RELEASE: 3.0.8-0.11-ppc64
VERSION: #1 SMP Thu Nov 10 16:28:46 UTC 2011 (3cea58b)
MACHINE: ppc64 (3550 Mhz)
MEMORY: 4 GB
PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log for details)
PID: 5563
COMMAND: "sh"
TASK: c0000000faac3700 [THREAD_INFO: c0000000f8ce0000]
CPU: 36
STATE: TASK_RUNNING (PANIC)
crash>
可以看到內核版本是 3.0.8-0.11-ppc64，這是一個 sles11sp2 的開發版本，
- 接下來，我們用 bt 命令來看一下堆疊
清單 7. bt 命令
crash> bt
PID: 5563 TASK: c0000000faac3700 CPU: 36 COMMAND: "sh"
#0 [c0000000f8ce31b0] .crash_kexec at c0000000001039f8
#1 [c0000000f8ce33b0] .die at c000000000020158
#2 [c0000000f8ce3450] .bad_page_fault at c000000000045004
#3 [c0000000f8ce34d0] handle_page_fault at c000000000005ec8
Data Access error [300] exception frame:
R0: 0000000000130000 R1: c0000000f8ce37c0 R2: c000000000f876d8
R3: c000000001224dc8 R4: 0000000000000001 R5: 0000000000000000
R6: cfffffffffffffff R7: 0000000002220000 R8: 2ffffffff1f10000
R9: d00000000e0f0000 R10: 0000000000000000 R11: 0000000100000000
R12: 0000000082002424 R13: c000000001f06c00 R14: 000000001003e270
R15: 0000000000000001 R16: 0000000000000001 R17: 0000000000000000
R18: 0000000000000000 R19: c0000000f820b4b8 R20: c0000000f8ce3df8
R21: c000000000fe2400 R22: 00000fffb53d0000 R23: fffffffffffff000
R24: 0000000000000400 R25: 000000000000ed99 R26: 0000000000002000
R27: 0000000000002e58 R28: c000000001224dc8 R29: c000000001224dc0
R30: c000000000ef2658 R31: c0000000f8ce39a0
NIP: c000000000255900 MSR: 8000000000009032 OR3: c000000000005278
CTR: c000000000263a08 LR: c0000000002558dc XER: 0000000000000001
CCR: 0000000022002444 MQ: 0000000000000001 DAR: 0000000100000008
DSISR: 0000000040000000 Syscall Result: 0000000000000000
.....
#4 [c0000000f8ce37c0] .get_vmalloc_info at c000000000255900
[Link Register ] [c0000000f8ce37c0] .get_vmalloc_info at c0000000002558dc (un
reliable)
#5 [c0000000f8ce3850] .meminfo_proc_show at c000000000263ad8
#6 [c0000000f8ce3b40] .seq_read at c00000000020aa44
#7 [c0000000f8ce3c30] .proc_reg_read at c000000000258ccc
#8 [c0000000f8ce3ce0] .vfs_read at c0000000001dee60
#9 [c0000000f8ce3d80] .sys_read at c0000000001df06c
#10 [c0000000f8ce3e30] syscall_exit at c0000000000097ec
syscall [c01] exception frame:
R0: 0000000000000003 R1: 00000ffff3cceb60 R2: 00000fffb5305c40
R3: 0000000000000008 R4: 00000fffb53d0000 R5: 0000000000000400
R6: 0000000000000001 R7: 00000fffb5249f88 R8: 800000000200f032
R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000
R12: 0000000000000000 R13: 00000fffb50b8110
NIP: 00000fffb523d0c4 MSR: 800000000200f032 OR3: 0000000000000008
CTR: 00000fffb51dae70 LR: 00000fffb51daeac XER: 0000000000000001
CCR: 0000000044002422 MQ: 0000000000000001 DAR: 00000fffb51dcd60
DSISR: 0000000040000000 Syscall Result: 00000fffb53d0000
Crash>
- 我們看到系統崩潰前的最后一個呼叫是“#4 [c0000000f8ce37c0] .get_vmalloc_info at c000000000255900”，現在用 dis 命令來看一下該地址的反匯編結果
清單 8. dis 命令
crash> dis -l c000000000255900
/usr/src/debug/kernel-ppc64-3.0.8/linux-3.0/fs/proc/mmu.c: 47
0xc000000000255900 <.get_vmalloc_info+112>: ld r10,8(r11)
- 從上面的反匯編結果中，我們看到問題出在 mmu.c 第 47 行代碼，翻開 linux 原始碼的相應位置
清單 9. linux 原始碼
21 void get_vmalloc_info(struct vmalloc_info *vmi)
22 {
23 struct vm_struct *vma;
……
46 for (vma = vmlist; vma; vma = vma->next) {
47 unsigned long addr = (unsigned long) vma->addr
用 struct 命令查看資料結構
清單 10. struct 命令
crash> struct -o vm_struct
struct vm_struct {
[0] struct vm_struct *next;
[8] void *addr;
[16] long unsigned int size;
[24] long unsigned int flags;
[32] struct page **pages;
[40] unsigned int nr_pages;
[48] phys_addr_t phys_addr;
[56] void *caller;
}
SIZE: 64
crash>
對照原始碼和反匯編代碼，我們發現第 47 行的原始碼，實際對應的就是反匯編的代碼
ld r10,8(r11) # 將暫存器 r11 的第 8 個 byte 后的內容，load 到暫存器 r10
- 那么 r11 中應該是 vm_struct 結構，我們再用 struct 來看看
清單 11. struct 命令
crash> struct vm_struct 0000000100000000
struct: invalid kernel virtual address: 0000000100000000
crash>
說明 r11 的內容已經被破壞，并不是指向一個 vm_struct 結構了，
經過上面的層層分析，我們推測問題的產生程序如下：mmu.c 第 46 行， vma = vma->next 取到了一個錯誤的地址，導致第 47 行 addr = (unsigned long) vma->addr 產生了內核錯誤，當然，更深層的原因，還需要對代碼邏輯進行分析，找出導致這個現象的根源，
常見問題
本節列出了使用 crash 程序中可能會碰到的問題，并給出了相應的解決對策，
- 缺少除錯資訊包
清單 12. 缺少除錯資訊包
[root@bondlp1 2012-02-02-01:37]# crash
crash 5.1.8-1.el5
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: /boot/vmlinuz-2.6.18-307.el5: no debugging data available
crash: vmlinuz-2.6.18-307.el5.debug: debuginfo file not found
crash: either install the appropriate kernel debuginfo package, or
copy vmlinuz-2.6.18-307.el5.debug to this machine
遇到這種問題時，需要安裝內核除錯資訊包，再重新運行 crash 命令，
- vmlinux 和 vmcore 版本不匹配
清單 13. vmlinux 和 vmcore 版本不匹配
[root@bondlp1 2012-02-02-01:37]# crash /usr/lib/debug/lib/modules/2.6.18-305.el5/vmlinux vmcore
crash 5.1.8-1.el5
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...
WARNING: kernel version inconsistency between vmlinux and dumpfile
please wait... (gathering module symbol data)
WARNING: cannot access vmalloc'd module memory
crash: invalid kernel virtual address: 8000000000b663c8 type: "runqueues entry (per_cpu)"
這種情況說明你所使用的 vmlinux 與產生 vmcore 的內核版本不一致，需要使用相同版本的內核來除錯 vmcore 檔案，
- core 檔案不完整
清單 14. core 檔案不完整
[root@bondlp1 2012-02-02-01:37]# crash /usr/lib/debug/lib/modules/2.6.18-305.el5/vmlinux vmcore
crash 5.1.8-1.el5
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmcore: may be truncated or incomplete
PT_LOAD p_offset: 1610681124
p_filesz: 234881024
bytes required: 1845562148
dumpfile size: 1638400000
這個提示說明你使用的 vmcore 檔案不完整，導致這個問題的原因可能有多種，硬碟空間不足，網路 dump 時網路中斷等等，對于這種情況，我們需要重新 dump 一個完整的 vmcore 進行分析除錯，
小結
對于內核開發人員，crash 已經成了必不可少的一個工具，內核固然高深，但是通過 kdump 和 crash 這對戰友的親密配合，很多問題都會迎刃而解，本文僅為您介紹了 crash 的基本知識，更多的技巧還需要讀者在實踐中不斷探索和總結，
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/1798.html
標籤：嵌入式
上一篇：樹莓派4 64bit 編譯安裝QT5.13.2 和 Redis Desktop Manager 2020.1-dev
下一篇：痞子衡嵌入式：恩智浦i.MX RT1xxx系列MCU啟動那些事（11.A）- FlexSPI NOR啟動時間(RT1170)