How to Capture Xen Hypervisor and Kernel Messages using a Serial Cable
How to capture the Xen hypervisor and dom0 kernel messages using a serial port
Background
Some Xen bugs, e.g. crashing virtual machines, or Xen itself crashing, require capturing the hypervisor log messages. Capturing the log messages typically entails connecting the troublesome Xen machine, via serial cable, to a second machine running a serial communication program such as minicom. This article describes how to configure Xen and dom0 to output log messages to a serial port. Both grub1 and grub2 configuration are covered. A minicom configuration quickstart is also provided.
Grub1
For systems using grub1 (e.g. SLE11 family), add a "Xen Debug" entry to the GRUB menu defined in /boot/grub/menu.lst. An example entry on a SLES11 SP4 Xen host would be
title Xen Debug -- com1
root (hd0,1) kernel /boot/xen.gz vga=mode-0x317 crashkernel=512M@16M dom0_mem=4096M console=com1 com1=115200,8n1 loglvl=all guest_loglvl=all module /boot/vmlinuz-3.0.101-88.1-xen root=/dev/disk/by-id/ata-INTEL_SSDSC2BA200G3_BTTV314102LW200GGN-part2 quiet splash resume=/dev/disk/by-id/ata-INTEL_SSDSC2BA200G3_BTTV314102LW200GGN-part1 splash=silent showopts console=hvc0 vga=0x317 module /boot/initrd-3.0.101-88.1-xen
To ensure any existing Xen or kernel parameters are preserved, it is suggested to make a copy of the existing Xen entry. On the copied entry, change the title, add 'console=com1 com1=115200,8n1 loglvl=all guest_loglvl=all' to the Xen parameters, and add 'console=hvc0' to the dom0 kernel parameters.
The saved changes to /boot/grub/menu.lst will take affect on next boot.
Grub2
In grub2, the serial port configuration information is defined in /etc/default/grub. The Xen parameters are appended to GRUB_CMDLINE_XEN_DEFAULT, e.g.
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem4096M console=com1 com1=115200,8n1 loglvl=all guest_loglvl=all"
The dom0 kernel parameters are appended to GRUB_CMDLINE_LINUX_XEN, e.g.
GRUB_CMDLINE_LINUX_XEN="console=hvc0"
After saving changes to /etc/default/grub, regenerate the grub configuration by running 'grub2-mkconfig -o /boot/grub2/grub.cfg'. The changes will take affect on next boot.
minicom configuration
Although typically not installed by default, minicom is available in all SUSE distros. Install with 'zypper install minicom'. To configure minicom, do the following:
From a terminal, run "minicom -s". Select "Serial port setup" and press Enter. Change "Serial Device" to "/dev/ttyS0" Change "Bps/Par/Bits" to "115200 8N1" Exit the menu by pressing Enter. Select "Save setup as dfl" so that these changes become persistent. Exit from Minicom.
Capturing log messages
Start minicom on the capture machine by running 'minicom -C filename', filename specifies where the captured log messages will be saved. Reboot the troublesome Xen machine, selecting the "Xen Debug" entry if using grub1. Once the bug is reproduced, close the capture file by exiting minicom (ctrl+a x). Capture file containing hypervisor and dom0 log messages can then be attached to a bugzilla entry.
In some cases, debugging may necessitate gathering domain, event, or register information from the hypervisor. Such system state is gathered using Xen's debug keys. To gain interactive access to the hypervisor console, press 'ctrl+a' six times. You will receive a message stating that input has been switched to Xen. Press 'h' to see the availabe debug keys and a short description of their function. Press the desired debug key to gather information requested by a bug assignee. E.g. to collect state information of domains press the 'q' key. (Note: return to dom0 console by pressing 'ctrl+a' six times. In other words, pressing 'ctrl+a' six times toggles between the dom0 and hypervisor consoles.)
When all requested log messages and debug key output has been collected, exit minico (ctrl+a x) and attach the capture file to a bugzilla entry.
Footnotes
If the Xen machine under investigation does not have a serial port, the only option is using a PCI serial card. USB-Serial will not work.