Backup

MySQL MYD and TMD files

Posted in Backup, WordPress on April 8th, 2008 by Johan Huysmans – 3 Comments

I just recovered from a database issue.

While running a mysqldump of my databases I received an error. Running a check on that specific database told me that the comments.MYD file was not found.

Investigating on the system showed me that due to an error the filesystem containing /var/lib/mysql was mounted read-only. Running a fsck on that specific filesystem solved that error but didn’t fix my corrupted table.

phpMyAdmin showed that the table was in use and while performing a check or repair it complained about the missing file.

The comments.MYD file was indeed missing, but there was a comments.TMD file, which wasn’t there for all the other tables.

Moving this TMD file to MYD solved the issue. I could repair the table and the check is again successful.

During this problem wordpress didn’t complain but it just didn’t show my comments. But I have no clue what caused this problem.

rsync daemon init script

Posted in Backup, Linux SysAdmin on December 21st, 2007 by Johan Huysmans – 1 Comment

Rsync, the popular file synchronization tool, can be used as a daemon. It is quite easy to start the daemon version of it, just execute:
rsync --daemon --config=/path/to/rsyncd.conf

If you want to start it at boottime you have to create an init script as it is not delivered with the rpm (or at least not with redhat/centos/fedora rpms).
You can search for any existing init scripts, for example systemimager-server delivers such init script, and use that script or write a simple init script yourself, using an existing script as guideline.

In my setup I needed 2 daemons, one running on port 873 (the default), the other one running on port 874. This is off course perfectly feasible but had some problems with the init scripts.
I wanted to be able to start and stop each daemon separately. But with the simple init script I could start both separately but when i wanted to stop one of them they are both stopped.

Diving into the daemon and killproc functions, which are stored in /etc/rc.d/init.d/functions and used by all init scripts, showed me that I have to use a combination of storing the pid file and the base filename of the init script.
The pid of the rsync daemon can be stored in any file (this is handle by rsync itself), this can be configured in the rsyncd configuration with following line: pid file = /var/run/rsyncd.pid

Following init script can start or stop the specific rsync daemon even if there are multiple daemons running on the system.
There is only requirement for this init script, which is explicitly explained in the script, the filename of the initscript must be the basename of the pidfile. For example if your initscript is /etc/init.d/custon-rsyncd the pid must be stored in /var/run/custom-rsyncd.pid.

#! /bin/sh
#
# chkconfig:   2345 50 50
# description: The rsync daemon

# source function library
 . /etc/rc.d/init.d/functions

PROG='/usr/bin/rsync'
BASE=${0##*/}

# Adapt the --config parameter to point to your rsync daemon configuration
# The config file must contain following line:
#  pid file = /var/run/<filename>.pid
# Where <filename> is the filename of the init script (= this file)
OPTIONS="--daemon --config=/etc/rsyncd.conf"

case "$1" in
  start)
    echo -n $"Starting $BASE: "
    daemon --check $BASE $PROG $OPTIONS
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$BASE
    echo
    ;;
  stop)
    echo -n $"Shutting down $BASE: "
    killproc $BASE
    RETVAL=$?
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$BASE
    echo
    ;;
  restart|force-reload)
    $0 stop
    sleep 1
    $0 start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|force-reload}" >&2
    exit 1
    ;;
esac

exit 0

Updated full backup script

Posted in Backup on August 19th, 2007 by Johan Huysmans – Be the first to comment

This week I updated the full backup page. We jumped from version 0.4 to 0.8.
Thanks to my colleagues: Raf, Fred and Jos the script is updated to meet our requirements.

Here are some of the major features added to the newest version:

  • You can choose if you want to keep a history (= full weekly backups and daily incrementals) or only 1 backup.
  • All information will be written to a logfile which can be mailed to some addresses when the script is done (previously this was done by an extra command in the cron).
  • After the backup we will check the size of the local data and the size of the backup, printed to the logfile.
  • We will check the free space on the backupdisk and print it to the logfile.

The behavior of the script can be configured with the variables at the top of the script.

My dump script now includes Allrights

Posted in Backup on January 4th, 2007 by Johan Huysmans – Be the first to comment

A new version (v0.6) is now available for download and can be found in the scripts section of my blog (link is at the top of every page).

Besides the dumping of a ldap, mysql and postgresql database and creating a dump of the rpm-package list and the partition table, it is now possible to dump all ownerships and permissions of all files.

This is done by an external script: allrights.pl. More information of this script can be found here.

There are also some minor improvements in the script: The db_dump directory is automatically created and the temporary directory is moved to /tmp and is deleted at the end of the script.

Any comments, remarks, feature-request, success-stories, … are welcome. I appreciate every feedback.

Dump script

Posted in Backup on November 2nd, 2006 by Johan Huysmans – Be the first to comment

I updated my dump script.

2 features are added:
* dump a package list of the rpm database (command: rpm -qa)
* dump the partition table (command: sfdisk -d)

The different dump-commands are now placed in a “if” structure, this means you can define at the top if you want to run that specific dump or not. The next thing I will change is to let the script decide you can dump something or not. So if the specific dump command exists it will run it otherwise it will just skip that dump-section.

You can find the dump script here. More Information about the dumping of the partition table can be found on MDLog:/sysadmin.

Mysql dump script

Posted in Backup on June 13th, 2006 by Johan Huysmans – Be the first to comment

Due to some problems we needed to restore the backup of 1 mysql database. I’ve got my mysql dump script installed on that system so that won’t be any problem.
Think again… There are 37 mysql database, with a total of almost 300Meg data. This is stored in one single gziped file of 30Meg. It isn’t easy to find the correct database.

Let’s fix this… We want 1 .tar.gz file containing the dumps of all database, every database is stored in a seperate file.
With the command mysqlshow you get a list of all database in mysql. The command mysqldump with the databasename as an argument will dump only that database.

Put this all together and you have version 0.4 of my mysqldump script.

Problems with an old rsync

Posted in Backup on May 4th, 2006 by Johan Huysmans – Be the first to comment

When installing my backup script on a redhat 9 i got this fatal error:

remote ip : Connection refused
rsync: connection unexpectedly closed (0 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)

It seems that I was working with old version of rsync (rsync version 2.5.7 protocol version 26). That old version uses default rsh, the recent versions use ssh. And ssh is what i need.
There is an option, called rsh, that allows you to set the remote shell to ssh.

# rsync --rsh=ssh ....

backup script

Posted in Backup on March 10th, 2006 by Johan Huysmans – Be the first to comment

Since several weeks I’m working with backups.

I created a script that dumps databases. It currently only supports mysql, ldap and postgresql. The script is not difficult, it just uses the dump program of the database and writes that to a file.

I created an other script that does the real backup. It uses rsync to copy the data to the other host, together with ssh. With some simple commands over ssh it will keep a history. It also will create a daily incremental backup linked to the weekly full backup.

http://www.raskas.be/blog/scripts/database-backup/ will bring you to the information about the database script.

http://www.raskas.be/blog/scripts/full-backup/ will bring you to the information about the backup script.

I made those script available online so everybody can use and improve those scripts. If you improved the script, please let me know, maybe I will update my scripts with your modifications.