CORE ANALYSIS SUITE
The core analysis suite is a self-contained tool that can be used to investigate either live systems, kernel core dumps created from the Kernel Core Dump patch offered by Mission Critical Linux, or kernel core dumps created by the LKCD patch.
- The tool is loosely based on the SVR4 crash command, but has been completely integrated with gdb in order to be able to display formatted kernel data structures, disassemble source code, etc.
- The current set of available commands consist of common kernel core analysis tools such as a context-specific stack traces, source code disassembly, kernel variable displays, memory display, dumps of linked-lists, etc. In addition, any gdb command may be entered, which in turn will be passed onto the gdb module for execution.
- There are several commands that delve deeper into specific kernel subsystems, which also serve as templates for kernel developers to create new commands for analysis of a specific area of interest. Adding a new command is a simple affair, and a quick recompile adds it to the command menu.
- The intent is to make the tool independent of Linux version dependencies, building in recognition of major kernel code changes so as to adapt to new kernel versions, while maintaining backwards compatibility.
These are the current prerequisites:
- At this point, Intel-, Alpha-, PowerPC-, and IA64-based kernels are supported. The IA64 version of crash has only been tested on live systems. Other architectures will be addressed in the future.
- One size fits all -- the utility can be run on any Linux kernel version from 2.2.5-15 through 2.4.*.
- In order to contain debugging data, the top-level kernel Makefile's CFLAGS definition must be modified, and the kernel rebuilt. In 2.2 kernels, change the following line:
CFLAGS = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
- to
CFLAGS = -Wall -Wstrict-prototypes -O2 -g
In 2.4 kernels, change the following line:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing
- to
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -g -fno-strict-aliasing
The deletion of -fomit-frame-pointer is done for dependable stack
traces.
- After the kernel is re-compiled, the uncompressed "vmlinux" kernel that is created in the top-level kernel build directory must be saved.
To build this utility, simply uncompress the tar file, enter the crash3.3 subdirectory, and type "make". The initial build will take several minutes because the gdb module must be configured and and built.
If the tool is run against a crash dumpfile, two arguments are required, the uncompressed kernel name and the core dumpfile name.
If run on a live system, only the kernel name is required, because /dev/mem will be used as the "dumpfile". If the kernel file is stored in /boot, /, /boot/efi, or in any /usr/src subdirectory, then no command line arguments are required -- the first kernel found that matches /proc/version will be used as the namelist.
For example, if the command name "crash" is kept intact in the Makefile, invoking it on a live system would look like this:
$ crash
crash 3.3
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 5.1
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"..
KERNEL: /boot/vmlinux
DUMPFILE: /dev/mem (live system)
CPUS: 1
DATE: Fri Feb 22 10:34:21 2002
UPTIME: 10 days, 22:55:18
LOAD AVERAGE: 0.08, 0.03, 0.01
TASKS: 42
NODENAME: ha2.mclinux.com
RELEASE: 2.4.0-test10
VERSION: #11 SMP Thu Nov 4 15:09:25 EST 2000
MACHINE: i686 (447 MHz)
MEMORY: 128 MB
PID: 3621
COMMAND: "crash"
TASK: c463c000
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> help
- files mod runq union alias foreach mount search vm ascii fuser net set vtop bt gdb p sig waitq btop help ps struct whatis dev irq pte swap wr dis kmem ptob sym q eval list ptov sys exit log rd task extend mach repeat timer
crash version: 3.3 gdb version: 5.1 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". For the most recent version: http://www.missioncriticallinux.com/download
crash>
When run on a dumpfile, both the kernel namelist and dumpfile must be entered on the command line, as in:
$ crash vmlinux.17 lcore.cr.17
crash 3.3
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 5.1
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux.17
DUMPFILE: lcore.cr.17
CPUS: 1
DATE: Wed Nov 10 19:54:47 1999
UPTIME: 00:02:02
LOAD AVERAGE: 0.82, 0.34, 0.12
TASKS: 43
NODENAME: ha3.mclinux.com
RELEASE: 2.2.5-15
VERSION: #39 Wed Nov 10 17:43:16 CST 1999
MACHINE: i686 (501 MHz)
MEMORY: 96 MB
PANIC: "tulip_interrupt"
PID: 286
COMMAND: "in.rlogind"
TASK: c0b3a000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
When run on an LKCD core dump, the command line would be like so:
$ crash vmlinux vmdump.0
The tool's environment is context-specific. On a live system, the default context is the command itself; on a dump the default context will be the task that panicked. The most commonly-used commands are:
set - set a new task context by pid, task address, or cpu.
bt - backtrace of the current context, or as specified with arguments.
p - print the contents of a kernel variable.
rd - read memory, which may be either kernel virtual, user virtual, or
physical.
ps - simple process listing.
log - dump the kernel log_buf.
struct - print the contents of a structure at a specified address. foreach - execute a command on all tasks, or those specified, in the system.
Detailed help concerning the use of each of the commands in the menu above may be displayed by entering "help command", where "command" is one of those listed above. Rather than getting bogged down in details here, simply run the help command on each of the commands above. Note that many commands have multiple options so as to avoid the proliferation of command names.
Command output may be piped to external commands or redirected to files. Enter "help output" for details.
The command line history mechanism allows for command-line recall and command-line editing. Input files containing a set of crash commands may be substituted for command-line input. Enter "help input" for details.
Note that a .crashrc file (or .<your-command-name>rc if the name has been changed), may contain any number of "set" or "alias" commands -- see the help pages on those two commands for details.
Lastly, if a command is entered that is not recognized, it is checked against the kernel's list of variables, structure, union or typedef names, and if found, the command is passed to "p", "struct", "union" or "whatis". That being the case, as long as a kernel variable/structure/union name is different than any of the current commands.
(1) A kernel variable can be dumped by simply entering its name:
crash> init_mm
init_mm = $2 = {
mmap = 0xc022d540,
mmap_avl = 0x0,
mmap_cache = 0x0,
pgd = 0xc0101000,
count = {
counter = 0x6
},
map_count = 0x1,
mmap_sem = {
count = {
counter = 0x1
},
waking = 0x0,
wait = 0x0
},
context = 0x0,
start_code = 0xc0000000,
end_code = 0xc022b4c8,
end_data = c0250388,
...
(2) A structure or can be dumped simply by entering its name and address:
crash> vm_area_struct c5ba3910
struct vm_area_struct {
vm_mm = 0xc3ae3210,
vm_start = 0x821b000,
vm_end = 0x8692000,
vm_next = 0xc5ba3890,
vm_page_prot = {
pgprot = 0x25
},
vm_flags = 0x77,
vm_avl_height = 0x4,
vm_avl_left = 0xc0499540,
vm_avl_right = 0xc0499f40,
vm_next_share = 0xc04993c0,
vm_pprev_share = 0xc0499060,
vm_ops = 0x0,
vm_offset = 0x0,
vm_file = 0x0,
vm_pte = 0x0
}
The crash utility has been designed to facilitate the task of adding new commands. New commands may be permanently compiled into the crash executable, or dynamically added during runtime using shared object files.
To permanently add a new command to the crash executable's menu:
- For a command named "xxx", put a reference to cmd_xxx() in defs.h.
- Add cmd_xxx into the base_command_table[] array in global_data.c.
- Write cmd_xxx(), putting it in one of the appropriate files. Look at the other commands for guidance on getting symbolic data, reading memory, displaying data, etc...
- Recompile and run.
Note that while the initial compile of crash, which configures and compiles the gdb module, takes several minutes, subsequent re-compiles to do such things as add new commands or fix bugs just takes a few seconds.
Alternatively, you can create shared object library files consisting of crash command extensions, that can be dynamically linked into the crash executable during runtime or during initialization. This will allow the the same shared object to be used with subsequent crash releases without having to re-merge the command's code into each new set of crash sources. The dynamically linked-in commands will automatically show up in the crash help menu. For details, enter "help extend" during runtime, or enter "crash -h extend" from the shell command line.
Complete documentation regarding this utility can be accessed by visiting:
http://oss.missioncriticallinux.com/projects/crash
From the page above you can join the Crash Mailing List, which we invite you to use for reporting bugs, making suggestions, asking questions, or sending any new commands you'd like to see permanently incorporated. Alternatively, you can send the same type of queries to:
support@missioncriticallinux.com