Kdump

From openSUSE

What is Kdump and how does it work?

Overview

Kdump is used to take kernel dumps. If a kernel has crashed it's very desirable to copy the memory image of the crashed environment to debug it afterwards to finally (hopefully) find the cause of the problem. In user space, that technique is called “core dumps” and GDB can be used to debug them.

Before Kdump was available, LKCD (an external kernel patch and related userspace tools) was used for kernel dumping. That used the crashed kernel to save the dump which had the disadvantage that dumping is likely to fail if the environment is undefined.

As Kexec is used to load a new kernel from an existing one, Kdump works quite similar: A 2nd kernel called “capture kernel” or “panic kernel” is executed after the first (running) kernel crashed. The difference is that in case of normal kexec, the 2nd kernel replaces the 1st kernel while in case of kdump the 2nd kernel is executed while still having access to the full memory of the 1st kernel. In the environment of the kdump kernel, the memory of the 1st kernel can be saved.

Background

The picture below shows the first 2 GiB of physical address space of a typical PC system.

When using Kdump, a specific amount (in that example: 64 MiB) are reserved for the crashkernel, i.e. “removed” from the system's memory. The user (or the tools of the distribution) must add the crashkernel=size@offset command line in the bootloader configuration to achieve this.

In that reserved area, the crashkernel will be loaded with the kexec utility, using the -p (panic) option. Now when the main kernel panics, the crashkernel will be executed from this memory. The special thing about that environment is that the usable memory is limited to the reserved amount. This is because the rest of the memory must be preserved and cannot be overwritten -- the content is needed because it now get saved.

The dump kernel (or better: the userspace running inside that dump kernel environment) can access the whole memory contents by

• the /dev/oldmem device in a linear manner, or
• the /proc/vmcore in the ELF core dump format.

After the contents of the old system has been saved, it must be rebooted in order to get the full working system back. kexec cannot be used to boot from the panic kernel into a “normal” kernel.

One special thing is that the kernel is not moved to its normal execution address (which would be for example 2 MiB on x86-64) before executing it. Because of this, the kernel must either be relocated to the execution address of the reserved memory are (on architectures that support this relocation and have CONFIG_RELOCATABLE enabled before compiling the kernel) or compiled with the matching CONFIG_PHYSICAL_START option. That's why special kernel-kdump packages are available.

Starting with openSUSE 10.3, only the ppc64 kernel has a special kdump counterpart. On the other architectures, namely i386, x86-64 and IA64, the relocation feature is used. On the SUSE Linux Enterprise Server 10 (no matter which service pack), the situation is different, only IA64 has a relocatable kernel there and the other architectures have kernel-kdump.

Usage and Configuration

Manual Usage

This describes the manual usage of kdump, without any distribution support. You only need to have the kexec-tools package installed. It can be obtained from [1].

1. Add the crashkernel=size@offset option to your bootloader configuration. See the table below for recommended options for the different architectures. Then reboot.
2. Load the panic kernel with kexec -p vmlinuz --append="command_line" --initrd="initrd". The command line should include the root file system (root=), the irqpoll and reset_devices options and should end with the runlevel you would like to have in the kdump environment (preferably 1).
3. Now crash the kernel, for testing you can use Sysrq-c.
4. In the kdump environment, copy /proc/vmcore away.
5. Reboot.

Manual Configuration

In openSUSE and SUSE Linux Enterprise Server, you have to perform following steps to enable kdump on each boot:

1. Add the crashkernel option to bootloader configuration and reboot. See above for a description of the values.
2. Enable the kdump init script with chkconfig kdump on or use the YaST runlevel editor.
3. You can edit the options in /etc/sysconfig/kdump. See the comments, /usr/share/doc/packages/kdump/README.SUSE or /usr/share/doc/packages/kexec-tools/README.SUSE for a descriptions of that options.
4. Now execute the init script once with rckdump start (or reboot).
5. To test, use the Sysrq-c. Now the dump should be saved automatically to the path KDUMP_SAVEDIR. After this, the system should automatically reboot if KDUMP_IMMEDIATE_REBOOT is on.

YaST configuration

You have to install the yast2-kdump package first because that's not in the default pattern. After this, start the “Kernel kdump” module in the “System” category, or just enter yast2 kdump on command line (as root).

Just select “Enable kdump”. The default value for the reserved memory should be ok in most situations. Then go to the “Dump Target” page and enter you preferred dump target.

Advances Topics

Dump Filtering

Normal kernel dumps are large: As much bytes as your system has memory plus some small overhead (per-CPU state and the usual ELF header). Copying such large files over network can be tedious, even with a fast internet connection and using compression like gzip or bzip2.

To debug kernel problems, some of the memory contents is not needed:

• zero pages
• userspace pages
• buffer pages
• free pages

The makedumpfile utility reads a ELF core dump (e.g. /proc/vmcore), filters out the pages and writes the result either as ELF core dump again or in a special “kdump compressed format”. The advantage is that the latter format can compress the dumps per page: In contrast to compressing the whole dump with gzip or bzip, you don't have to extract the dump before using it: crash can read the compressed pages and decompresses the pages on demand.

Debugging

See Crashdump_Debugging for more information about debugging crash dumps.

References

• Kdump, A Kexec-based Kernel Crash Dumping Mechanism
• Crash Whitepaper