Category Archives: Xen

Using Xen for High Availability Clusters

ONLAMPYesterday evening O’Reilly published “Using Xen for High Availability Clusters“.

This article is written by my colleague, Kris Buytaert, and me and describes the setup of the clustering of the virtual gateways at Newtec.
It explains the networking in a xen dom0 and how to block all virtual machines from the network. It also explains how you can integrate this with linux-ha.

This is my first article on O’Reilly, this means that I now have a profile on O’Reillynet, you can check it out here:

Enjoy the reading !

3ware module refuses to load with xen kernel

Today I encountered a problem with the 3ware module.

The fresh installed CentOs 5, on a machine with a 3ware 9500 card, booted without any problems, but this wasn’t the case when booting with the xen enabled kernel. During boot of the xen kernel, which ended in a kernel panic, it showed something about ioremap. When googling about the problem I found a patch which solved the problem.

Here is a step-by-step guide how I’ve applied the patch and solved the problem… Continue reading

Beryl and Java apps

While using Beryl as my default window manager, I noticed that some java applications didn’t work correctly, for example the XenServer Client.
The application started correctly but I only saw a gray screen.
Whenever my Metacity window manager (that’s the one of gnome) was active, instead of Beryl, the application works as it was meant to be.

This problem is explained in the beryl wiki.
The solution is also mentioned there.

Because I’m using java 5 (1.5.0u11 from Sun) the fix is very simple.
I added following line at the end of /etc/bashrc.

export AWT_TOOLKIT=MToolkit

After restarting the X server the java applications works without any problems.

Less than minimal

Last week I updated my CentOS base image document.
One important change was in the yum line.

Previously it would do a groupinstall of Base, which results in a total of 300 packages. This is the same amount of packages when it is installed from cd and minimal install is checked.
But this minimal install includes way to much and useless packages like pcmcia, irda, isdn, …
The groupinstall of Core gives a better result. Now only a bit more than 100 packages are installed. Even yum or openssh aren’t installed. So you will have to add some extra packages but at least you’re not stuck with all those unused packages and running services.

The same happens when you kickstart. By default it will install the base and core groups. But again this results in 300 packages. Just mentioning core or not mentioning base in the packages section doesn’t solve the problem.
Luckily there is an, undocumented, option for the packages section. You can pass –nobase if you don’t want to install the complete Base group. But now you will have to mention the Core group or it will install not enough packages.
This is how your packages section in the kickstart file can look like:

% packages --nobase
@ Core

Multi Bridge Xen

Kris explained in a previous blog post how you can create multiple bridges.

While testing the multiple bridges in a situation where every bridge is connected to a physical interface and that every virtual interface must be connected to the correct bridge (peth0->xenbr0->vifX.0; peth1->xenbr1->vifX.1; …) I noticed something strange.

The vif interface was not always connected to the correct bridge. When the bridgename is provided in the configuration file (with or without the mac address) the first entry is not always mapped to vifX.0.

A wrapper script brought again a solution. The vif-wrapper-bridge script will chech the name of the vif interface and use the last number (interface number) to define the bridge. That bridgename is stored in the xenstore and used by the vif-bridge script.

[root@GW-Y ~]$ grep wrapper xend-config.sxp
(network-script network-wrapper-bridge)
(vif-script vif-wrapper-bridge)

if [ $1 = "online" ]
  # load some general functions
  dir=$(dirname "$0")
  . "$dir/"
  # find the bridge number out of the vif interface name
  brnum=$(echo $vif | sed 's/vif.*\.//')
  # store the bridgename in xenstore
  bridge=$(xenstore_write "$XENBUS_PATH/bridge" "$bridge")
# load the real vif-bridge script
/etc/xen/scripts/vif-bridge $1

Xen dom0 and heartbeat

During testing of heartbeat 1 on a xen dom0, some strange errors appeared. One of the error messages was:
ERROR: No local heartbeat. Forcing shutdown

This error message was explained in the heartbeat FAQ but the explained causes didn’t make any sense.
An update from Xen 3 to Xen 3.0.4 didn’t solve the problem.
Updating heartbeat from v1 to v2 made the error disappear and heartbeat was now working without any problems.
Heartbeat v2 has a complete new config file. The main config file ( is now an xml file. But don’t worry, if you don’t want to upgrade the config file, you can keep working with a v1 config file. You can’t take advantage of the new features of heartbeat v2 but at least you’re taking advantage of the bugfixes ;)

Xen kernel-panic

I repeadetly tried to install xen. The installation of the rpms, downloaded from the xen site, succeeded but when I rebooted into the xen kernel a kernel panic occured.
A message like no version for “struct_module” found: kernel tainted scrolled over the screen during the loading of the modules. A deeper look into that message showed me that it was not fatal and the kernel panic was caused by something else.

A lot of kernel panics are caused because they can’t find the harddisk. This happens because the correct module isn’t build into the kernel or available in the initrd. The same problem occured here. The xen kernel has not as many build in modules as a normal centos kernel and the needed modules aren’t automatically added in the initrd.

The solution is easy: include the needed modules in the initrd.
But which module is the correct one? That depends on the hardware in your system. You can start finding the correct modules by reading the dmesg when you boot that system with a working kernel.

Following command worked on my home machine:

[root@xen ~]# mkinitrd -v -f --with=ide-generic /boot/initrd-

Using a serial link on a xen dom0

By default xen will bind it’s console on ttyS0 and following message is visible in /var/log/message
kernel: Xen virtual console successfully installed as ttyS0

At this moment you can’t send messages over a serial cable connected to your system because the device is already in use by xen.

Adding following parameter in the grub config file, the default ttyS0 will be changed to whatever you provide or removed if you enter xencons=off.
module /boot/vmlinuz-2.6-xen ro root=/dev/sda1 xencons=ttyS9

ttyS0 will be unused after a reboot and the module providing the serial devices can be loaded, the module is called 8250.
Normally the serial modules are compiled in the kernel and nothing is implemented to load them during boot. But we want to load when the system is booted. Therefore I added that module in /etc/modprobe.conf:
# loading the module for serial devices. We want this at boot time therefore it is aliased to snd-card.
alias snd-card-0 8250
alias snd-card-0 8250_pci
alias snd-card-0 8250_pnp

This is how you can test the serial connection between 2 machine:
* On machine 1 open the serial device: cat /dev/ttyS0
* On machine 2 send a message over the link: echo “Hello World” > /dev/ttyS0
The message should appear on machine 1.