Until we upgraded to heartbeat v3.0.3 and pacemaker v1.1.5.
When we tried to swap the resource group depending on the drbd disk, the resource group was stopped but the drbd master / slave resource refused to demote the current master.
Resulting in the unavailability of the resource group.

Mentioning this problem on #linux-ha they pointed me to resource-stickiness in the configuration.

In our configuration the resource-stickiness is set to INFINITY, we have configured it like that because we only want resources to swap during a failure or when manually requested, in all other cases the resource should remain on the node where it currently is. This is not the case when the resource-stickiness is not configured.

Apparently setting it to INFINITY was a bit radical, setting the resource-stickiness to 1000 is sufficient. The resources only swaps during a failure or manually triggered and the drbd master / slave resources follows the resource group using it.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
ea:34:84:3f:f3:1e:74:78:7e:f7:5f:4d:a3:5d:3d:9f.
Please contact your system administrator.
Add correct host key in /home/johan/.ssh/known_hosts to get rid of this message.
Offending key in /home/johan/.ssh/known_hosts:187
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
X11 forwarding is disabled to avoid man-in-the-middle attacks.
Permission denied (publickey,gssapi-with-mic,password).

After this message you have to remove line 187 from your .ssh/known_hosts file and perform the ssh command again.

If you want to get rid of this message and the altering of the known_hosts file you can add following ssh options to the .ssh/config file:

StrictHostKeyChecking no
UserKnownHostsFile /dev/null


Apt told me that it depends on a specific file which isn’t provided by any package in the repository. After some investigation I noticed that the rpm WAS available in the repository, and that yum correctly found that package.

The problem was that the specific file needed by the package was a symlink provided by the other package. The symlink file is known by the rpm (rpm -ql /path/to/file gives the rpm) but isn’t know by apt.

Instead of running genbasedir with the location of the repository as only argument, add the –bloat argument.

genbasedir --bloat /path/to/repository

This will solve the problem!

I noticed this problem during the installation of redhat-lsb on a very minimal CentOS5 system. redhat-lsb requires 2 files (which are symlinks), these files are provided by… redhat-lsb itselve…

If you didn’t use the –bloat argument the redhat-lsb package couldn’t be installed with apt, it could be installed with yum or rpm.

We are using syslog-ng-2.1.3 on one of our machines which sends part of his messages over UDP to 2 syslog machines. On some days we noticed that syslog-ng and some other services are stopped. Restarting syslog-ng showed us that they were killed by the OOM-killer.
I directly suspected the java process that was also running on that machine.

After googling around I found this syslog-ng bug: https://bugzilla.balabit.com/show_bug.cgi?id=39

And indeed, we had the same problem. This is how I could reproduce it:

• Stop the syslog service (on the host which receives the messages)
• Restart syslog-ng
• Watch the memory usage of syslog-ng growing until it starts swapping and triggers the OOM-killer

Luckily this bug is already solved, and by upgrading to syslog-ng-2.1.4 the problem is fixed.

CentOS doesn’t provide the rpm packages of syslog-ng, silfreed.net does: http://www.silfreed.net/download/repo/

How come appending of both STDOUT and STDERR to a file does not work this way? Is this a bug in bash?
Yes, I know I can use the following, but I prefer the &>>:
ls 1>> output_and_error.txt 2>&1


And how come I can’t find a bugzilla for bash?
Yes, I know I can use the command bashbug to send a mail to a mailinglist, but this is not the same as bugzilla.

One of the big annoyancies I noticed during the installation was that it didn’t recognize my lvm partitions. And I really need that, as my home and root partition are on lvm and I didn’t want to repartition my complete drive.

Luckily I found some explanation for lvm support during the installation. This is the summary of the actions you need to perform.

Become the root user:
ubuntu@ubuntu:~$ sudo -i

Load the dm-mod module:
root@ubuntu:~# modprobe dm-mod

Install the lvm2 package on the live system:
root@ubuntu:~# apt-get install lvm2

Activate the logical volumes of your volumegroup
root@ubuntu:~# lvchange -a y <volgroup name>

At this moment you can perform a normal installation, your existing logical volumes will be recognized and can be used during installation.
After the complete installation process you have to install lvm support for you new installation.

Mount the partitions of your new installation:
root@ubuntu:~# mount /dev/volgroup/root /mnt
root@ubuntu:~# mount /dev/sda1 /mnt/boot
root@ubuntu:~# mount -o bind /dev /mnt/dev

Chroot into your new installation:
root@ubuntu:~# chroot /mnt

Install the lvm2 package:
root@ubuntu:~# apt-get install lvm2

You can now exit your chroot environment and reboot your machine. If everything is OK, you can boot your fresh ubuntu installation.

[johan@laptop ~]$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -nr|head
201 ls
108 ssh
108 cd
98 sh
81 su
29 vi
24 rpm
24 ping
23 rm
20 mv

[johan@workstation ~]$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -nr|head
149 ls
128 cd
125 cvs
93 diff
71 echo
54 ssh
45 for
28 mkdir
26 md5sum
24 reset

My previous post of my Top 10 used commands was more than 1 year ago.

While running a mysqldump of my databases I received an error. Running a check on that specific database told me that the comments.MYD file was not found.

Investigating on the system showed me that due to an error the filesystem containing /var/lib/mysql was mounted read-only. Running a fsck on that specific filesystem solved that error but didn’t fix my corrupted table.

phpMyAdmin showed that the table was in use and while performing a check or repair it complained about the missing file.

The comments.MYD file was indeed missing, but there was a comments.TMD file, which wasn’t there for all the other tables.

Moving this TMD file to MYD solved the issue. I could repair the table and the check is again successful.

During this problem wordpress didn’t complain but it just didn’t show my comments. But I have no clue what caused this problem.

This article is written by my colleague, Kris Buytaert, and me and describes the setup of the clustering of the virtual gateways at Newtec.
It explains the networking in a xen dom0 and how to block all virtual machines from the network. It also explains how you can integrate this with linux-ha.

This is my first article on O’Reilly, this means that I now have a profile on O’Reillynet, you can check it out here: http://www.oreillynet.com/pub/au/3323.

Enjoy the reading !

The fresh installed CentOs 5, on a machine with a 3ware 9500 card, booted without any problems, but this wasn’t the case when booting with the xen enabled kernel. During boot of the xen kernel, which ended in a kernel panic, it showed something about ioremap. When googling about the problem I found a patch which solved the problem.

Here is a step-by-step guide how I’ve applied the patch and solved the problem…

Ideal you download the src rpm of the kernel, apply the patch and rebuild the kernel. This rebuild will take some time, but you can easily download the 3ware module and rebuild only that module. Therefore I’ve used the module provided on the 3ware website.

1. Install the needed packages to be able to build the module

# yum install kernel-xen-devel gcc

2. Create a link pointing to the source directory of the kernel (This is the kernel where you want to rebuild the module for). This is not needed if you want to rebuild for the current kernel.

# cd /usr/src/
# ln -snf /lib/modules/2.6.18-53.1.4.el5xen/source linux

3. Download and unpack the downloaded module

# tar -zxvf 3w-9xxx-2.6.18kernel_9.5.0.tgz
# cd build

4. Apply the patch (see http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg10302.html)

5. Build the module

# make

6. Overwrite the old 3ware module with the one you just have build

# cp 3w-9xxx.ko /lib/modules/2.6.18-53.1.4.el5xen/kernel/drivers/scsi/3w-9xxx.ko

7. Rebuild the initrd

# mkinitrd -v -f /boot/initrd-2.6.18-53.1.4.el5xen.img 2.6.18-53.1.4.el5xen

8. Reboot