This article is written by my colleague, Kris Buytaert, and me and describes the setup of the clustering of the virtual gateways at Newtec.
It explains the networking in a xen dom0 and how to block all virtual machines from the network. It also explains how you can integrate this with linux-ha.

This is my first article on O’Reilly, this means that I now have a profile on O’Reillynet, you can check it out here: http://www.oreillynet.com/pub/au/3323.

Enjoy the reading !

The fresh installed CentOs 5, on a machine with a 3ware 9500 card, booted without any problems, but this wasn’t the case when booting with the xen enabled kernel. During boot of the xen kernel, which ended in a kernel panic, it showed something about ioremap. When googling about the problem I found a patch which solved the problem.

Here is a step-by-step guide how I’ve applied the patch and solved the problem…

Ideal you download the src rpm of the kernel, apply the patch and rebuild the kernel. This rebuild will take some time, but you can easily download the 3ware module and rebuild only that module. Therefore I’ve used the module provided on the 3ware website.

1. Install the needed packages to be able to build the module

# yum install kernel-xen-devel gcc

2. Create a link pointing to the source directory of the kernel (This is the kernel where you want to rebuild the module for). This is not needed if you want to rebuild for the current kernel.

# cd /usr/src/
# ln -snf /lib/modules/2.6.18-53.1.4.el5xen/source linux

3. Download and unpack the downloaded module

# tar -zxvf 3w-9xxx-2.6.18kernel_9.5.0.tgz
# cd build

4. Apply the patch (see http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg10302.html)

5. Build the module

# make

6. Overwrite the old 3ware module with the one you just have build

# cp 3w-9xxx.ko /lib/modules/2.6.18-53.1.4.el5xen/kernel/drivers/scsi/3w-9xxx.ko

7. Rebuild the initrd

# mkinitrd -v -f /boot/initrd-2.6.18-53.1.4.el5xen.img 2.6.18-53.1.4.el5xen

8. Reboot

This problem is explained in the beryl wiki.
The solution is also mentioned there.

Because I’m using java 5 (1.5.0u11 from Sun) the fix is very simple.
I added following line at the end of /etc/bashrc.

export AWT_TOOLKIT=MToolkit

After restarting the X server the java applications works without any problems.

Previously it would do a groupinstall of Base, which results in a total of 300 packages. This is the same amount of packages when it is installed from cd and minimal install is checked.
But this minimal install includes way to much and useless packages like pcmcia, irda, isdn, …
The groupinstall of Core gives a better result. Now only a bit more than 100 packages are installed. Even yum or openssh aren’t installed. So you will have to add some extra packages but at least you’re not stuck with all those unused packages and running services.

The same happens when you kickstart. By default it will install the base and core groups. But again this results in 300 packages. Just mentioning core or not mentioning base in the packages section doesn’t solve the problem.
Luckily there is an, undocumented, option for the packages section. You can pass –nobase if you don’t want to install the complete Base group. But now you will have to mention the Core group or it will install not enough packages.
This is how your packages section in the kickstart file can look like:

% packages --nobase
@ Core
yum
openssh-clients
openssh-server


While testing the multiple bridges in a situation where every bridge is connected to a physical interface and that every virtual interface must be connected to the correct bridge (peth0->xenbr0->vifX.0; peth1->xenbr1->vifX.1; …) I noticed something strange.

The vif interface was not always connected to the correct bridge. When the bridgename is provided in the configuration file (with or without the mac address) the first entry is not always mapped to vifX.0.

A wrapper script brought again a solution. The vif-wrapper-bridge script will chech the name of the vif interface and use the last number (interface number) to define the bridge. That bridgename is stored in the xenstore and used by the vif-bridge script.


[root@GW-Y ~]$ grep wrapper xend-config.sxp
(network-script network-wrapper-bridge)
(vif-script vif-wrapper-bridge)


vif-wrapper-bridge
#!/bin/sh
 
if [ $1 = "online" ]
then
  # load some general functions
  dir=$(dirname "$0")
  . "$dir/vif-common.sh"
  # find the bridge number out of the vif interface name
  brnum=$(echo $vif | sed 's/vif.*\.//')
  bridge=xenbr$brnum
  # store the bridgename in xenstore
  bridge=$(xenstore_write "$XENBUS_PATH/bridge" "$bridge")
fi
 
# load the real vif-bridge script
/etc/xen/scripts/vif-bridge $1


This error message was explained in the heartbeat FAQ but the explained causes didn’t make any sense.
An update from Xen 3 to Xen 3.0.4 didn’t solve the problem.
Updating heartbeat from v1 to v2 made the error disappear and heartbeat was now working without any problems.
Heartbeat v2 has a complete new config file. The main config file (ha.cf) is now an xml file. But don’t worry, if you don’t want to upgrade the config file, you can keep working with a v1 config file. You can’t take advantage of the new features of heartbeat v2 but at least you’re taking advantage of the bugfixes

A lot of kernel panics are caused because they can’t find the harddisk. This happens because the correct module isn’t build into the kernel or available in the initrd. The same problem occured here. The xen kernel has not as many build in modules as a normal centos kernel and the needed modules aren’t automatically added in the initrd.

The solution is easy: include the needed modules in the initrd.
But which module is the correct one? That depends on the hardware in your system. You can start finding the correct modules by reading the dmesg when you boot that system with a working kernel.

Following command worked on my home machine:

[root@xen ~]# mkinitrd -v -f --with=ide-generic /boot/initrd-2.6.16.33-xen_3.0.4.1.img 2.6.16.33-xen_3.0.4.1


At this moment you can’t send messages over a serial cable connected to your system because the device is already in use by xen.

Adding following parameter in the grub config file, the default ttyS0 will be changed to whatever you provide or removed if you enter xencons=off.
module /boot/vmlinuz-2.6-xen ro root=/dev/sda1 xencons=ttyS9

ttyS0 will be unused after a reboot and the module providing the serial devices can be loaded, the module is called 8250.
Normally the serial modules are compiled in the kernel and nothing is implemented to load them during boot. But we want to load when the system is booted. Therefore I added that module in /etc/modprobe.conf:
# loading the module for serial devices. We want this at boot time therefore it is aliased to snd-card.
alias snd-card-0 8250
alias snd-card-0 8250_pci
alias snd-card-0 8250_pnp


This is how you can test the serial connection between 2 machine:
* On machine 1 open the serial device: cat /dev/ttyS0
* On machine 2 send a message over the link: echo “Hello World” > /dev/ttyS0
The message should appear on machine 1.