Xen3 yet another cross OS Integration Howto in OpenSuSE 10.3 Xen3 host
From openSUSE
Introduction
In this howto I will explain how to setup an OpenSuSE 10.3 Xen3 host with different guests like OpenSuSE, Debian-Etch, CentOS and Windows. Another goal of this howto is to migrate existing systems or VMware guests in to Xen.
Hardware configuration of the OpenSuSE 10.3 Xen3 host (dom0)
AMD Athlon(tm) 64 X2 Dual Core Processor 4200+, 2 GB of RAM and 1 nic (Asrock ALiveNF7G-HD720 Mainboard). Check if your AMD 64 CPU has Pacifica support (HVM fullvirtualization). You can use a Live-CD or DVD for this purpose (SuSE Live). After booting into the Live SuSE System issue the following command:
grep "svm" /proc/cpuinfo
This command should return a list of the CPU features that your processor has. If it doesn't then your CPU is not HVM capable. You can only run paravirtualized guests on your system.
Note that I was not able to use my 3ware sata-raid controller with the Xen kernel. I did not find a workaround and had to remove it. The symptoms were (iommu) error messages in my system logs. Check if your Hardware (Raid controller) is compatible with the Xen 3.1.0_15042-51.3 kernel!
Installation of the OpenSuSE 10.3 Xen host (dom0)
Install Suse 10.3 (x86-64) with Gnome or KDE. It might be good idea to setup the system with LVM.
Use the community repositories in the Yast software section to setup repositories. Haupt-Repository (DEBUG), Haupt-Repository (Nicht-OSS), Haupt-Repository (Opensource Software - OSS), Haupt-Repository (Quellen), Haupt-Update Repository, Packman Repository, openSUSE BuildService - Virtualisierung (QEMU), openSUSE BuildService - Virtualisierung (VirtualBox). Not all of the listed repositories are needed for Xen.
After setting up the repositories install the following packages: (xen, Version: 3.1.0_15042-51.3), (kernel-xen, Version: 2.6.22.17-0.1), (xen-libs, Version: 3.1.0_15042-51.3), (xen-tools, Version: 3.1.0_15042-51.3), (xen-tools-ioemu, Version: 3.1.0_15042-51.3), (yast2-vm, Version: 2.16.1-48).
UPDATE your system via Yast online update.
Start Yast --> System --> Virtualization --> Install Hypervisor and Tools and follow the remaining installation steps, and do not reboot, we still have to do some modifications.
Adapt the SuSEfirewall2 (thats placed in /etc/sysconfig/SuSEfirewall2), to route traffic. Here just the sections that you have to modify for Xen setup.
FW_ROUTE="yes" FW_CONFIGURATIONS_EXT="sshd" FW_FORWARD="0/0,0/0" FW_FORWARD_ALWAYS_INOUT_DEV="xenbr+")
I have only one nic in my system, this is why I had to open the sshd port.
For production use its better if the dom0's network hardware is redundant via 4 nic's in a Ethernet Bonding setup connected to 4 switches. The best if you have 4 nic's in your server. The first 2 nics configured via Bonding and connected to the production LAN. The second 2 nic's also configured via Bonding and connected to the management (administration) LAN.
Now modify Grub to default to Xen and reboot.
UPDATE your system via Yast online update.
Use the Yast gui when installing new guests and change the guest configuration later manually for your own needs.
SuSEfirewall2 and Xen networking: You will notice that after starting a guest system there are a lot of martian source log messages in your dom0 system logs (var/log/messages). The only way worked out for me in conjunction with the SuSEfirewall2 was to modify the SuSEfirewall2 to suppress these messages. Note that these messages could be security relevant (its a kind of monitoring of wrong addressed or impossible packets, that could be an indicator for malicious traffic e.g. packet manipulation, spoofing or an attack), but in my case it comes from the system routing i set up for Xen. And since we will isolate the dom0 system from the server network and never start any network services in it, the modification is not risky. Here the workaround. I wrote a function for the SuSEfirewall (/sbin/SusEfirewall2). Edit the file (/sbin/SuSEfirewall2) and add the function (suppress_log_martians) just under the last funktion (drop_all), dont modify the function (drop_all). These are the relevant sections, first the function definition and second the function call :
1 -function definition -->
dom0:/ # cat /sbin/SuSEfirewall2
....
# Special Logging + Deny #
drop_all()
{
local chain
....
done
}
# Saman Behnam
# Suppress Martian source Kernel messages because of the Xen networking configuration
suppress_log_martians()
{
for MARTIANS in /proc/sys/net/ipv4/conf/*/log_martians; do
echo 0 > $MARTIANS
done
}
############################################
# #
# Now we begin to set the filter rules ... #
# #
############################################
....
2 -function call --> place the function call just after the call of (handle_initscripts) and before the (# END #) mark. Like the in following section:
handle_initscripts # Saman Behnam suppress_log_martians # END # die 0 "Firewall rules successfully set" # vim: sw=4
Netloop devices: Due to the Xen virtual networking every time a domu reboots it will connect to a new higher numbered vif device, e.g. once you have connected to vif[x] you cannot use it again. Thus you can run out of netloop devices. You can see that behavior by issuing (brctl show) after every reboot of a domu. You can increase your system netloop devices.
dom0:/ # cat /etc/modprobe.conf.local # # please add local extensions to this file # options netloop nloopbacks=64 options loop max_loop=32 dom0:/ #
Setup the sysclock of the domus independent from dom0. See http://en.opensuse.org/Xen3_yet_another_Virtual_Network_Concept .
Clustering (DRBD, Heart-Beat) should be possible for Xen hosts and guests.
Also a shared storage with 2 dom0's for live migration purposes and fast failover reaction should be taken in consideration.
Installation of the Xen guest operating systems (domu)
Before we start with the installation of the guests, here are some tips that apply to all guests (domu's)
Loop devices: Note when attaching drives to your domu's via the (file:) option e.g. (disk=[ 'file:/data1/xen/images/debian-base/disk0,hda,w', 'file:/data1/etch-64-netinstall.iso,hdc:cdrom,r', ]) it will use a free loop device in your system. You can check this by issuing (losetup -a). At default there are 8 free loop devices in your system. You can also increase the loop devices (/etc/modprobe.conf.local --> options loop max_loop=32 ). But the best is if you use the (tap:aio:) option e.g. (tap:aio:=[ 'file:/directory/to/xen/images/mydomu/disk0,hda,w', 'tap:aio::/directory/etch-64-netinstall.iso,hdc:cdrom,r', ]) in the domu's config file when attaching drives to domu's, supposed your kernel provides tap support and the Suse kernel does. Note that Yast will create the domu's with the (file:) option per default. You should change it manually to (tap:aio:) afterwards.
Connecting to guests in dom0: You can check the port that VNC is using after starting a domu in dom0 by issuing (netstat -tulpen | grep 59) in dom0, that will display a list of the ports that are used by the guests. You can also specify a port to connect to it after a domu is started. This can be done by changing the line (vfb=['type=vnc,vncunused=1']) to (vfb=['type=vnc,vnclisten=127.0.0.1,vncdisplay=11']) in the domu's config file. This example will start the VNC console of the domu on 127.0.0.1:5911 in dom0. I prefer to use SSH or to connect to the virtual serial console of the domu by issuing (xm console domu-name). If you use kdm or gdm then a good choice would be Freenx. Use the packages from the Nomachine home page, they just work fine.
CentOS5.1 x86-64 paravirtualized:
Yast --> Virtualization --> Create Virtual Mashines. Choose rh5 template, choose paravirtualized installation and begin install from bootable DVD install media image (CentOS DVD has support for paravirtual boot and installation).
CDROM hard disk (image) driver problem: During installation the CentOS installer will ask for a CDROM driver and is not able to continue installation from media. Strong guess is that the CentOS installer does not load the xennet and xenblk drivers to detect the xvd[x] xen virtual drive. I didn't try to install it from real DVD e.g. from physical DVD drive :) ! Choose a network installation source and continue installation and finish. Installation and boot of the new system works out of the box so far. The new system does not need a Grub bootloader, this is done from within the dom0 via the domuloader.
After updating or installing a new kernel, we have to point to the new name of the kernel in the domu config file in dom0 e.g.
bootargs="--entry=xvdb1:/vmlinuz-2.6.18-53.1.14.el5xen,/initrd-2.6.18-53.1.14.el5xen.img"
Bootloaders inside the guest does not work! This is my CentOS domu config file that is located in (/etc/xen/vm):
dom0:/etc/xen/vm # cat centos5.1-x86-64-1-pv # name="centos5.1-x86-64-1-pv" ostype="rhel5" uuid="9b8795bc-10ff-e43b-4ab8-3dfc95566e7d" memory=256 vcpus=2 on_crash="destroy" on_poweroff="destroy" on_reboot="restart" localtime=0 builder="linux" bootloader="/usr/lib/xen/boot/domUloader.py" bootargs="--entry=xvdb1:/vmlinuz-2.6.18-53.1.14.el5xen,/initrd-2.6.18-53.1.14.el5xen.img" # After update kernel name is 2.6.18-53.1.14.el5, so we have to rename here also e.g. vmlinuz-2.6.18-53.1.14.el5xen initrd-2.6.18-53.1.14.el5xen.img extra=" " #disk=[ 'file:/data1/CentOS-5.1-x86_64-bin-DVD.iso,xvda:cdrom,r', 'tap:aio:/data1/xen/images/centos5.1-x86-64-1/disk0,xvdb,w', ] disk=[ 'phy:/dev/cdrom,hdc:cdrom,r', 'tap:aio:/data1/xen/images/centos5.1-x86-64-1-pv/disk0,xvdb,w', ] vif=[ 'mac=00:16:3e:7b:30:2a', ] vfb=['type=vnc,vnclisten=127.0.0.1,vncdisplay=10'] #vfb=['type=vnc,vncunused=1'] #boot="d" dom0:/etc/xen/vm #
Note that I have changed the bootargs section after installing a new kernel. I also changed the disk section from the (file:) option to the (tap:aio) option to not exhaust the systems loop devices.
Opensuse 10.3 (x86-64) paravirtualized:
Worked out of the box. Even after kernel update.
Opensuse 10.3 (x86) paravirtualized:
Worked out of the box. Even after kernel update.
Debian Etch (x86-64) paravirtualized:
Debian does have very good Xen and virtualization support but I was not successful to start an automatic installation with the Debian Etch media in paravirtualized mode! Instead I started a base net installation in fullvirtualized mode (choose OS --> other in Yast). I installed the whole system into one partition and did not use LVM in this simple installation. If you want LVM you have to clone the system. The cloning method can also be used to migrate any existing system to a virtualized mashine, supposed there is an official Xen Kernel for that distribution. Also if your CPU doesn't support HVM (AMD Pacifica or Intel VT) e.g. if you cannot start the installation in fullvirtualized mode then the cloning method is a good choice. I will explain the cloning method later. I also installed Grub to the MBR, we need a system boot loader in fullvirtualized mode! The installation works out of the box. I call the system debian-base. Additionally I installed SSH and VIM.
After finishing installation the domu configuration file that was created by Yast looks like this:
dom0:~# cat /etc/xen/vm/debian-base # name="debian-base" ostype="other" uuid="396fb776-a977-7dfb-2d05-8d727fe7b820" memory=256 vcpus=1 on_crash="destroy" on_poweroff="destroy" on_reboot="restart" localtime=0 builder="hvm" extid=0 device_model="/usr/lib/xen/bin/qemu-dm" kernel="/usr/lib/xen/boot/hvmloader" boot="c" disk=[ 'file:/data1/xen/images/debian-base/disk0,hda,w', 'file:/data1/etch-64-netinstall.iso,hdc:cdrom,r', ] vif=[ 'mac=00:16:3e:1a:b2:94,model=rtl8139,type=ioemu', ] stdvga=0 vnc=1 vncunused=1 apic=0 acpi=1 pae=1 usb=1 usbdevice='tablet' serial="pty"
Now we convert the fullvirtualized Debian domu to paravirtualized.
Shutdown the system and make a backup copy of the config file and the disk of your Debian domu.
Start the domu (xm create debian-base) and login to it. My domu disk configuration looks like this:
debian-base:~# fdisk -l
Disk /dev/xvda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/xvda1 * 1 116 931738+ 83 Linux
/dev/xvda2 117 130 112455 5 Extended
/dev/xvda5 117 130 112423+ 82 Linux swap / Solaris
Disk /dev/xvdb: 6 MB, 6426624 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/xvdb doesn't contain a valid partition table
debian-base:~#
Also we have to edit some system files (/etc/fstab, /etc/mtab/, /etc/modules, /boot/grub/device.map, /boot/grub/menu.lst) to get the system booting in a fully paravirtualized mode.
fstab before
debian-base:~# cat /etc/fstab # /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/hda1 / ext3 defaults,errors=remount-ro 0 1 /dev/hda5 none swap sw 0 0 /dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0 debian-base:~#
fstab after
debian-base:~# cat /etc/fstab # /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/xvda1 / ext3 defaults,errors=remount-ro 0 1 /dev/xvda5 none swap sw 0 0 /dev/xvdb /media/cdrom0 udf,iso9660 user,noauto 0 0 debian-base:~#
mtab before
debian-base:~# cat /etc/mtab /dev/hda1 / ext3 rw,errors=remount-ro 0 0 tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0 proc /proc proc rw,noexec,nosuid,nodev 0 0 sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 procbususb /proc/bus/usb usbfs rw 0 0 udev /dev tmpfs rw,mode=0755 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0 debian-base:~#
mtab after
/dev/xvda1 / ext3 rw,errors=remount-ro 0 0 tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0 proc /proc proc rw,noexec,nosuid,nodev 0 0 sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 procbususb /proc/bus/usb usbfs rw 0 0 udev /dev tmpfs rw,mode=0755 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0 debian-base:~#
So any hd[xy] goes to xvd[xy].
/etc/modules before
debian-base:~# cat /etc/modules # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. loop debian-base:~#
/etc/modules after
debian-base:~# cat /etc/modules # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. loop xennet xenblk debian-base:~#
/boot/grub/device.map before
debian-base:~# cat /boot/grub/device.map (hd0) /dev/hda debian-base:~#
/boot/grub/device.map after
debian-base:~# cat /boot/grub/device.map (hd0) /dev/xvda debian-base:~#
/boot/grub/menu.lst (just the sections that i have modified !) before
debian-base:~# cat /boot/grub/menu.lst # kopt=root=/dev/hda1 ro ## ## End Default Options ## title Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-6-xen-amd64 root (hd0,0) kernel /boot/xen-3.0.3-1-amd64.gz module /boot/vmlinuz-2.6.18-6-xen-amd64 root=/dev/hda1 ro console=tty0 module /boot/initrd.img-2.6.18-6-xen-amd64 savedefault title Debian GNU/Linux, kernel 2.6.18-6-amd64 root (hd0,0) kernel /boot/vmlinuz-2.6.18-6-amd64 root=/dev/hda1 ro initrd /boot/initrd.img-2.6.18-6-amd64 savedefault title Debian GNU/Linux, kernel 2.6.18-6-amd64 (single-user mode) root (hd0,0) kernel /boot/vmlinuz-2.6.18-6-amd64 root=/dev/hda1 ro single initrd /boot/initrd.img-2.6.18-6-amd64 savedefault ### END DEBIAN AUTOMAGIC KERNELS LIST debian-base:~#
/boot/grub/menu.lst (just the sections that i have modified !) after
debian-base:~# cat /boot/grub/menu.lst # kopt=root=/dev/xvda1 ro ## ## End Default Options ## title Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-6-xen-amd64 root (hd0,0) kernel /boot/xen-3.0.3-1-amd64.gz module /boot/vmlinuz-2.6.18-6-xen-amd64 root=/dev/xvda1 ro console=tty0 module /boot/initrd.img-2.6.18-6-xen-amd64 savedefault title Debian GNU/Linux, kernel 2.6.18-6-amd64 root (hd0,0) kernel /boot/vmlinuz-2.6.18-6-amd64 root=/dev/xvda1 ro initrd /boot/initrd.img-2.6.18-6-amd64 savedefault title Debian GNU/Linux, kernel 2.6.18-6-amd64 (single-user mode) root (hd0,0) kernel /boot/vmlinuz-2.6.18-6-amd64 root=/dev/xvda1 ro single initrd /boot/initrd.img-2.6.18-6-amd64 savedefault ### END DEBIAN AUTOMAGIC KERNELS LIST debian-base:~#
Now we install the Xen kernel with apt (debian-base:~# apt-get install xen-linux-system-2.6.18-6-xen-amd64). The initrd image is created automatically. Now shutdown the system.
We make a new paravirtualized config file for the domu and change also the mac like follows.
dom0:~# cat /etc/xen/vm/debian-base # name="debian-base" ostype="other" uuid="396fb776-a977-7dfb-2d05-8d727fe7b820" memory=256 vcpus=2 on_crash="destroy" on_poweroff="destroy" on_reboot="restart" localtime=0 builder="linux" bootloader="/usr/lib/xen/boot/domUloader.py" bootargs="--entry=xvda1:/boot/vmlinuz-2.6.18-6-xen-amd64,/boot/initrd.img-2.6.18-6-xen-amd64" extra="root=/dev/xvda1 vga=0x31a console=tty0" vfb=['type=vnc,vnclisten=127.0.0.1,vncdisplay=16'] disk=[ 'tap:aio:/data1/xen/images/debian-base/disk0,xvda,w', 'tap:aio:/data1/etch-64-netinstall.iso,xvdb:cdrom,r', ] vif=[ 'mac=00:16:3e:1a:b3:94', ]
Look at the following 2 entries in the domu configuration. Sinse I have installed the whole system into one partition the kernel and the inird files are in the boot dir and I have to point to them like follows
(bootargs="--entry=xvda1:/boot/vmlinuz-2.6.18-6-xen-amd64,/boot/initrd.img-2.6.18-6-xen-amd64") (extra="root=/dev/xvda1 vga=0x31a console=tty0")
If you have installed your system with separate boot partition and a separate root partition then these entries should look like this:
(bootargs="--entry=xvda1:/vmlinuz-2.6.18-6-xen-amd64,/initrd.img-2.6.18-6-xen-amd64") (extra="root=/dev/xvda2 vga=0x31a console=tty0")
Now start the system (xm create debian-base) with the new paravirtualized configuration and connect to the system (by issuing xm console debian-base) and you will see the a new Xen nic is detected by the system. To get the domu network working again we have to adapt also (/etc/udev/rules.d/z25_persistent-net.rules).
/etc/udev/rules.d/z25_persistent-net.rules (before)
debian-base:~# cat /etc/udev/rules.d/z25_persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# MAC addresses must be written in lowercase.
# PCI device 0x10ec:0x8139 (8139cp)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:16:3e:1a:b2:94", NAME="eth0"
# Xen virtual device (vif)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:16:3e:1a:b3:94", NAME="eth1"
debian-base:~#
/etc/udev/rules.d/z25_persistent-net.rules (after)
debian-base:~# cat /etc/udev/rules.d/z25_persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# MAC addresses must be written in lowercase.
# PCI device 0x10ec:0x8139 (8139cp)
#SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:16:3e:1a:b2:94", NAME="eth0"
# Xen virtual device (vif)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:16:3e:1a:b3:94", NAME="eth0"
debian-base:~#
After a reboot the Debian domu has connection to the network. Thats all!
Windows XP x86 fullvirtualized:
Worked out of the box.
CentOS5.1 x86-64 fullvirtualized:
Although it is not recommended to install Xen capable guests in fullvirtualized mode, it is good to know that it worked out of the box.
Opensuse 10.3 (x86) fullvirtualized:
Although it is not recommended to install Xen capable guests in fullvirtualized mode, it is good to know that it worked out of the box.
Opensuse 10.3 (x86-64) fullvirtualized:
Although it is not recommended to install Xen capable guests in fullvirtualized mode, I wasn't successful installing Opensuse 10.3 in fullvirtualized mode (Error no bootsector found!). I had installed Opensuse 10.3 in VMware server as guest. So I migrated the VMware guest to Xen. The steps are just like follows:
Migrating from VMware to Xen
[1] :Prerequisites. I assume that you have a running SuSE10.3-x86-64 guest in Vmware server. Before starting with migration you should check the ata_piix SCSI module, that should be listed in etc/sysconfig/kernel (in the ... INITRD_MODULES= ... section). This module is also needed by XEN. To convert the VMware guest to XEN we need both VMware guest files, the (config) vmdk file (in my case SUSE_Linux_64-bit-0.vmdk) and the (disk image) flat.vmdk file (in my case SUSE_Linux_64-bit-0-flat.vmdk).
At first you need to convert the vmdk file to a flat vmdk file on the VMware mashine. I call it in this example (SUSE_Linux_64-bit-conv3.vmdk). Issue the following command pointing to the vmdk config file:
vmware-vdiskmanager -r SUSE_Linux_64-bit-0.vmdk -t 0 SUSE_Linux_64-bit-conv3.vmdk
This will create a new flat vmdk image file (SUSE_Linux_64-bit-conv3.vmdk
In the second step we will convert the flat vmdk image (SUSE_Linux_64-bit-conv3.vmdk) to a raw image, that is usable by Xen. Here the command:
qemu-img convert -f vmdk SUSE_Linux_64-bit-conv3.vmdk -O raw SUSE_Linux_64-bit-conv3.xen
The qemu-img command is a part of the qemu package. I have installed (qemu-0.9.0.cvs-35).
After doing the above steps you have to create a Xen fullvirtualized domu config file and start the (SUSE_Linux_64-bit-conv3.xen) in Xen. My config looks like this:
# name="suse10.3-x86-64-1-fv" ostype="other" uuid="36c7d936-626c-1ca0-efcb-f173a1ab33b0" memory=256 vcpus=2 on_crash="destroy" on_poweroff="destroy" on_reboot="destroy" localtime=0 builder="hvm" extid=0 device_model="/usr/lib/xen/bin/qemu-dm" kernel="/usr/lib/xen/boot/hvmloader" boot="c" disk=[ 'tap:aio:/data1/xen/images/suse10.3-x86-64-1-fv/SUSE_Linux_64-bit-conv3.xen,hda,w', 'tap:aio:/data1/openSUSE-10.3-GM-KDE-x86_64.iso,hdc:cdrom,r', ] vif=[ 'mac=00:16:3e:4b:75:a7,model=rtl8139,type=ioemu', ] stdvga=0 vnc=1 vncunused=1 apic=0 acpi=1 pae=1 serial="pty"
If you have problems booting the domu then check the number of the vcpus. The first boot after migration takes time ... just wait a little :). After booting a new nic is detected. So you have to adapt the networking stuff in (/etc/udev/rules.d/...net..persistent). See also the Debian section.
Also consider using LVM in domus to manage disk space without rebooting.
Migrating to XEN (The system cloning method)
The system cloning method is a very universal method and is always a good choice if you need a clone (running copy) of a running system without interrupting it. For example if you have to move a running production system to newer or faster and bigger hardware. Also if you need an exact copy of a running production system for testing new applications on it ... etc. It's basically targeted at Linux systems and also some other UNIX flowers.
In this example we have a running guest (debian1) in Vmware, that we want to clone to Xen without rebooting or stopping (For example if you need a clone of a production system for testing purposes). This is the configuration of the system: We can see here it's a x86 Debian-Etch
debian1:~# uname -a Linux debian1 2.6.18-4-k7 #1 SMP Wed May 9 23:42:01 UTC 2007 i686 GNU/Linux debian1:~#
debian1:~# fdisk -l Disk /dev/sda: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 61 489951 82 Linux swap / Solaris /dev/sda2 * 62 522 3702982+ 83 Linux debian1:~#
debian1:~# df -lh Dateisystem Größe Benut Verf Ben% Eingehängt auf /dev/sda2 3,5G 2,8G 571M 84% / tmpfs 126M 0 126M 0% /lib/init/rw udev 10M 48K 10M 1% /dev tmpfs 126M 0 126M 0% /dev/shm debian1:~#
debian1:~# ip addr show
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:0b:4d:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.36/24 brd 192.168.200.255 scope global eth0
inet6 fe80::20c:29ff:fe0b:4dc1/64 scope link
valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
debian1:~#
In dom0 create an empty disk image for the destination (the clone) system.
dom0:~ # cd /data1/xen/images/ dom0:/data1/xen/images # mkdir clone dom0:/data1/xen/images # cd clone dom0:/data1/xen/images/clone# dd if=/dev/zero bs=1M count=4000 >> disk0
Attach the empty image to the rescue system domu (rescue). I use here a paravirtualized SuSE10.3 x86-64 domu as a rescue system.
section of domu config before attaching the image file.
disk=[ 'tap:aio:/data1/os/openSUSE-10.3-GM-DVD-x86_64.iso,xvda:cdrom,r', 'tap:aio:/data1/xen/images/suse10.3-x86-64-1-pv/disk0,xvdb,w', ]
section of domu config after attaching the empty image.
disk=[ 'tap:aio:/data1/os/openSUSE-10.3-GM-DVD-x86_64.iso,xvda:cdrom,r', 'tap:aio:/data1/xen/images/suse10.3-x86-64-1-pv/disk0,xvdb,w', 'tap:aio:/data1/xen/images/clone/disk0,xvdc,w', ]
Boot the rescue domu.
xm create suse10.3-x86-64-1-pv2
After login into the rescue system we can see the attached disk as xvdc
rescue:~ # fdisk -l
Platte /dev/xvda: 4363 MByte, 4363677696 Byte
255 heads, 63 sectors/track, 530 cylinders
Units = Zylinder of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Festplatte /dev/xvda enthält keine gültige Partitionstabelle
Platte /dev/xvdb: 4294 MByte, 4294967296 Byte
255 heads, 63 sectors/track, 522 cylinders
Units = Zylinder of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0007169f
Gerät boot. Anfang Ende Blöcke Id System
/dev/xvdb1 1 32 257008+ 83 Linux
/dev/xvdb2 33 522 3935925 8e Linux LVM
Platte /dev/xvdc: 4194 MByte, 4194304000 Byte
255 heads, 63 sectors/track, 509 cylinders
Units = Zylinder of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Festplatte /dev/xvdc enthält keine gültige Partitionstabelle
Platte /dev/dm-0: 285 MByte, 285212672 Byte
255 heads, 63 sectors/track, 34 cylinders
Units = Zylinder of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
...
rescue:~ #
Now lets take a look at the modules that are loaded in our paravirtualized rescue domu. We need to include some of these modules into the initrd of our clone system.
rescue:~ # lsmod lsmod Module Size Used by ip6t_LOG 11136 8 nf_conntrack_ipv6 26112 4 xt_pkttype 6400 3 ipt_LOG 10752 9 xt_limit 7552 17 ip6t_REJECT 9984 3 xt_tcpudp 7808 8 ipt_REJECT 9216 3 xt_state 7040 8 iptable_mangle 7424 0 iptable_nat 12292 0 nf_nat 25388 1 iptable_nat iptable_filter 7552 1 ip6table_mangle 7296 0 nf_conntrack_ipv4 16528 6 iptable_nat nf_conntrack 71900 5 nf_conntrack_ipv6,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4 nfnetlink 10936 4 nf_conntrack_ipv6,nf_nat,nf_conntrack_ipv4,nf_conntrack ip_tables 24792 3 iptable_mangle,iptable_nat,iptable_filter ip6table_filter 7296 1 ip6_tables 18888 3 ip6t_LOG,ip6table_mangle,ip6table_filter x_tables 24584 11 ip6t_LOG,xt_pkttype,ipt_LOG,xt_limit,ip6t_REJECT,xt_tcpudp,ipt_REJECT,xt_state,iptable_nat,ip_tables,ip6_tables ipv6 311288 23 nf_conntrack_ipv6,ip6t_REJECT,ip6table_mangle apparmor 46384 0 8250 36888 0 serial_core 27520 1 8250 loop 24580 0 joydev 15616 0 ext3 144400 3 jbd 76776 1 ext3 mbcache 13960 1 ext3 linear 10240 0 dm_snapshot 21832 0 dm_mod 64864 11 dm_snapshot xenblk 23008 5 xennet 33544 0
Lets see which of these modules are listed in the system configuration to be built in into the initrd.
rescue:~ # grep -ir -E "(xennet | xenblk)" /etc /etc/init.d/rc5.d/K09xend: # - xenblk when xend prepares for bootloader /etc/init.d/rc5.d/K09xend: modprobe xenblk 2>/dev/null || true /etc/init.d/rc5.d/S13xend: # - xenblk when xend prepares for bootloader /etc/init.d/rc5.d/S13xend: modprobe xenblk 2>/dev/null || true /etc/init.d/xend: # - xenblk when xend prepares for bootloader /etc/init.d/xend: modprobe xenblk 2>/dev/null || true /etc/init.d/rc3.d/K09xend: # - xenblk when xend prepares for bootloader /etc/init.d/rc3.d/K09xend: modprobe xenblk 2>/dev/null || true /etc/init.d/rc3.d/S13xend: # - xenblk when xend prepares for bootloader /etc/init.d/rc3.d/S13xend: modprobe xenblk 2>/dev/null || true /etc/sysconfig/kernel:DOMU_INITRD_MODULES="xennet xenblk" /etc/rc.d/rc5.d/K09xend: # - xenblk when xend prepares for bootloader /etc/rc.d/rc5.d/K09xend: modprobe xenblk 2>/dev/null || true /etc/rc.d/rc5.d/S13xend: # - xenblk when xend prepares for bootloader /etc/rc.d/rc5.d/S13xend: modprobe xenblk 2>/dev/null || true /etc/rc.d/xend: # - xenblk when xend prepares for bootloader /etc/rc.d/xend: modprobe xenblk 2>/dev/null || true /etc/rc.d/rc3.d/K09xend: # - xenblk when xend prepares for bootloader /etc/rc.d/rc3.d/K09xend: modprobe xenblk 2>/dev/null || true /etc/rc.d/rc3.d/S13xend: # - xenblk when xend prepares for bootloader /etc/rc.d/rc3.d/S13xend: modprobe xenblk 2>/dev/null || true rescue:~ #
As we can see here, there is an interesting entry in /etc/sysconfig/kernel . Lets take a look there.
rescue:~ # cat /etc/sysconfig/kernel ## Path: System/Kernel ## Description: ## Type: string ## Command: /sbin/mkinitrd # # This variable contains the list of modules to be added to the initial # ramdisk by calling the script "mkinitrd" # (like drivers for scsi-controllers, for lvm or reiserfs) # INITRD_MODULES="xenblk processor thermal fan jbd ext3 dm_mod edd" ## Type: string ## Command: /sbin/mkinitrd # # This variable contains the list of modules to be added to the initial # ramdisk that is created for unprivilegd Xen domains (domU); you may need # drivers for virtual block and network devices in addition to filesystem # and device-mapper modules. # DOMU_INITRD_MODULES="xennet xenblk" ## Type: string ## ServiceRestart: boot.loadmodules # # This variable contains the list of modules to be loaded # once the main filesystem is active # You will find a few default modules for hardware which # can not be detected automatically. # MODULES_LOADED_ON_BOOT="" ... rescue:~ #
OK we can see here the (DOMU_INITRD_MODULES="xennet xenblk") and (INITRD_MODULES="xenblk processor thermal fan jbd ext3 dm_mod edd") entries, so we need the xennet and xenblk if running as domu. Thus we should include xennet and xenblk in the clone's initrd. I will do this in the later steps.
Start creating a FAT and a file system in the empty virtual disk (xvdc). I will also migrate the source system to LVM, so i will create a LVM also.
Creating partitions:
rescue:~ # fdisk /dev/xvdc Befehl (m für Hilfe): n Befehl Aktion e Erweiterte p Primäre Partition (1-4) p Partitionsnummer (1-4): 1 Erster Zylinder (1-509, Vorgabe: 1): Benutze den Standardwert 1 Letzter Zylinder oder +Größe, +GrößeK oder +GrößeM (1-509, Vorgabe: 509): +250M Befehl (m für Hilfe): n Befehl Aktion e Erweiterte p Primäre Partition (1-4) p Partitionsnummer (1-4): 2 Erster Zylinder (32-509, Vorgabe: 32): Benutze den Standardwert 32 Letzter Zylinder oder +Größe, +GrößeK oder +GrößeM (32-509, Vorgabe: 509): Benutze den Standardwert 509 Befehl (m für Hilfe): a Partitionsnummer (1-4): 1 Befehl (m für Hilfe): t Partitionsnummer (1-4): 2 Hex code (L um eine Liste anzuzeigen): 8e Der Dateisystemtyp der Partition 2 ist nun 8e (Linux LVM) Befehl (m für Hilfe): w Die Partitionstabelle wurde verändert!
Creating LVM:
rescue:~ # pvcreate /dev/xvdc2 Physical volume "/dev/xvdc2" successfully created rescue:~ # vgcreate clone /dev/xvdc2 Volume group "clone" successfully created rescue:~ # lvcreate -n root -L +3000M clone Logical volume "root" created rescue:~ # lvcreate -n swap -L +500M clone Logical volume "swap" created rescue:~ # lvcreate -n home -L +250M clone Rounding up size to full physical extent 252,00 MB Insufficient free extents (62) in volume group clone: 63 required rescue:~ # lvcreate -n home -L +200M clone Logical volume "home" created
Creating file system:
rescue:~ # mke2fs -j /dev/xvdc1
rescue:~ # mke2fs -j /dev/clone/root
rescue:~ # mke2fs -j /dev/clone/home
rescue:~ # mkswap /dev/clone/swap
Mount the attached hard disk (xvdc) into a file structure that reflects the filesystem of the clone system. Note that you have to follow the mount order ... e.g. mount the root partition before boot ...etc
rescue:~ # cd ~ rescue:~ # mkdir dest rescue:~ # mount /dev/mapper/clone-root dest/ rescue:~ # cd dest/ rescue:~/dest # mkdir boot rescue:~/dest # mkdir home rescue:~/dest # mkdir dev rescue:~/dest # mkdir sys rescue:~/dest # mkdir proc rescue:~/dest # mount /dev/xvdc1 boot/ rescue:~/dest # mount /dev/mapper/clone-home home/ rescue:~/dest # mount /sys/ sys/ --bind rescue:~/dest # mount /proc/ proc/ --bind rescue:~/dest # mount /dev/ dev/ --bind
Now I will copy the Linux file system from the source system into the clone system
rescue:~/dest # ssh root@192.168.200.36 "tar cfv - /bin /boot /cdrom /etc /home /initrd /initrd.img /lib /media /mnt /opt /root /sbin /selinux /srv /tmp /usr /var /vmlinuz" | tar xpvf -
Now we chroot to the clone system and start modifying it to run as a paravirtualized Xen guest.
rescue:~/dest # cd .. rescue:~ # chroot dest/
From now on we are in the clone system and at first we start modifying /etc/fstab
rescue:/# vim /etc/fstab
fstab (before)
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/sda2 / ext2 defaults,errors=remount-ro 0 1 /dev/sda1 none swap sw 0 0 /dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
fstab (after)
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/xvda1 /boot ext3 defaults 1 2 /dev/mapper/clone-root / ext3 defaults,errors=remount-ro 0 1 /dev/mapper/clone-swap none swap sw 0 0 /dev/mapper/clone-home /home ext3 defaults 0 1 /dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0
We have to modify /etc/mtab
rescue:/# vim /etc/mtab
mtab (before)
/dev/sda2 / ext2 rw,errors=remount-ro 0 0 tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0 proc /proc proc rw,noexec,nosuid,nodev 0 0 sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 procbususb /proc/bus/usb usbfs rw 0 0 udev /dev tmpfs rw,mode=0755 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0 rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
mtab (after)
/dev/mapper/clone-root ext3 rw,errors=remount-ro 0 0 /dev/xvda1 /boot ext3 rx 0 0 /dev/mapper/clone-home /home ext3 0 0 tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0 proc /proc proc rw,noexec,nosuid,nodev 0 0 sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 procbususb /proc/bus/usb usbfs rw 0 0 udev /dev tmpfs rw,mode=0755 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
Modify /etc/modules . Note that this step is hardware dependent. For Xen paravirtualized mode we need the xennet and xenblk drivers. If you want to clone to another system (e.g. physical system) you should check (lsmod) first in the clone (target) physical, virtualization or paravirtualization system, which drivers are needed for disk drive and Raid controllers or special file systems ... etc. These drivers will be then be built in into initrd file. In case of cloning a SuSE system you have to look at /etc/sysconfig/kernel . You should also be able to clone CentOS and Ubuntu systems with this method
rescue:/# vim /etc/modules
modules (before)
# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. loop
modules (after)
# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. loop xennet xenblk
We modify /boot/grub/device.map
rescue:/# vim /boot/grub/device.map
device.map (before)
(hd0) /dev/sda
device.map (after)
(hd0) /dev/xvda
We modify /boot/grub/menu.lst (although we don't need Grub in paravirtualized mode)
rescue:/# vim /boot/grub/menu.lst
I will show only the relevant sections!
menu.lst (before)
... # kopt=root=/dev/sda2 ro ... ## ## End Default Options ## title Debian GNU/Linux, kernel 2.6.18-4-k7 root (hd0,1) kernel /boot/vmlinuz-2.6.18-4-k7 root=/dev/sda2 ro initrd /boot/initrd.img-2.6.18-4-k7 savedefault title Debian GNU/Linux, kernel 2.6.18-4-k7 (single-user mode) root (hd0,1) kernel /boot/vmlinuz-2.6.18-4-k7 root=/dev/sda2 ro single initrd /boot/initrd.img-2.6.18-4-k7 savedefault ### END DEBIAN AUTOMAGIC KERNELS LIST
menu.lst (after)
... # kopt=root=/dev/mapper/clone-root ro ... ## ## End Default Options ## title Debian GNU/Linux, kernel 2.6.18-4-k7 root (hd0,1) kernel /vmlinuz-2.6.18-4-k7 root=/dev/mapper/clone-root ro initrd /initrd.img-2.6.18-4-k7 savedefault title Debian GNU/Linux, kernel 2.6.18-4-k7 (single-user mode) root (hd0,1) kernel /vmlinuz-2.6.18-4-k7 root=/dev/mapper/clone-root ro single initrd /initrd.img-2.6.18-4-k7 savedefault ### END DEBIAN AUTOMAGIC KERNELS LIST
Check wether LVM is installed or not. If not then install to get the clone system working with LVM.
rescue:/# apt-cache search lvm rescue:/# apt-get install lvm2
Install the Xen kernel via APT. This will also create the initrd file. This step is only needed because we need a new kernel for running in paravirtualized mode. If you intent to clone for example a physical system to Xen fullvirtualized or VMware, then you maight not need to install a new kernel. Anyway you must run mkinitramfs, mkinitrd, update-initrd ... etc. to get the initrd file build. Otherwise your clone system would not boot.
rescue:/# apt-cache search xen rescue:/# apt-get install xen-linux-system-2.6.18-6-xen-686
In this cloning step I usually install the boot loader Grub to the MBR of the first bootable partition but I am not going to do this here because its not needed in paravirtualization mode.
rescue:~ # grub-install /dev/xvda1
Now exit the chroot environment and shut down the rescue system.
rescue:/# exit exit rescue:~ # shutdown now -h
Create a new domu config file for the clone in dom0.
dom0:/etc/xen/vm # vim clone # This is fully paravirtualized name="debian-etch-x86-clone-pv" ostype="other" uuid="d1255a7e-74a9-aa45-2ed2-2f0da3829c9a" memory=256 vcpus=2 on_crash="destroy" on_poweroff="destroy" on_reboot="restart" localtime=0 builder="linux" bootloader="/usr/lib/xen/boot/domUloader.py" bootargs="--entry=xvda1:/vmlinuz-2.6.18-6-xen-686,/initrd.img-2.6.18-6-xen-686" extra="root=/dev/mapper/clone-root ro vga=0x31a console=tty0" vfb=['type=vnc,vnclisten=127.0.0.1,vncdisplay=25'] disk=[ 'tap:aio:/data1/xen/images/clone/disk0,xvda,w', 'tap:aio:/data1/os/debian-40r3-amd64-kde-CD-1.iso,hdc:cdrom,r', ] vif=[ 'mac=00:12:1e:2c:c2:eb', ]
Now boot the clone (xm create clone).
After booting a new nic is detected and we have to edit (/etc/udev/rules.d/z25_persistent-net.rules) to get the network running.
debian1:~# vim /etc/udev/rules.d/z25_persistent-net.rules
z25_persistent-net.rules (before)
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# MAC addresses must be written in lowercase.
# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:0b:4d:c1", NAME="eth
0"
# Xen virtual device (vif)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:12:1e:2c:c2:eb", NAME="eth1
As you can see here a new nic (Xen virtual device (vif)) is detekted. And we have to adapt the new nic.
z25_persistent-net.rules (after)
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# MAC addresses must be written in lowercase.
# PCI device 0x1022:0x2000 (pcnet32)
#SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:0b:4d:c1", NAME="eth
0"
# Xen virtual device (vif)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:12:1e:2c:c2:eb", NAME="eth0
Also the IP adress and the hostname should be adapted! (/etc/network/interfaces /etc/hostname, /etc/hosts).
Have fun!
In memory of Christian Haase, he was one of Germany’s best and dedicated Linux and OpenSource developers and a good friend of all that knew him.
Saman Behnam
Dipl. Ing. (FH)
sbehnam73@yahoo.de
Deshalb koennen Pinguine nicht fliegen! Was nicht fliegt, kann nicht abstuerzen.
This site uses the YAML CSS framework.
About openSUSE | Disclaimers | Feedback
This page was last modified 11:34, 25 May 2008. This page has been accessed 1,768 times.
