Back to files

####################################################################### # bkp Backup Scripts Package
#
# (c) 2004 Joe Jaworski 25-JUN-04 v0.5 # Email: jj_at_joejaworski.com
# bkp is covered under the GNU General License. See the # LICENSE file accompanying this distribution. #
#######################################################################

Preface

While writing these scripts, the ultimate disaster happened to me. The IDE hard drive containing the '/' root partition on my server failed. Without warning the server just died. All of the files were gone forever. Interestingly, the IDE drive itself was only about a year old and still under warranty. It had never been exposed to excessive shock, vibration, or other stresses. I could never recover any data from the drive. If you think you can get away without a backup solution, you're wrong.

I began writing these scripts because I couldn't find anything that was easy to use and modify to fit my needs. All I wanted to do was tape backup from a single server to an IDE tape drive. I looked at Amanda, Taper, Mondo, Flexbackup, and most others. All of them seemed too complicated and an overkill for what I wanted to do.

The "bare-metal" backup prorgams are a good idea, but they rely on CDROM burners and multiple disks. I wanted something that could be automated via cron. Besides, it's fairly easy to do a new install on a new hard drive (you might want to change partition sizes or other things the second time around). So I wrote this package with the intent to do a restore AFTER your system has had a fresh (minimal) install of your distro. I also did not want to rely on special restore programs to get my data back. Therefore, any backup made with bkp can be restored just be using tar with your tapes.

My situation is a SOHO environment with a single server. I needed to provide daily backups of the server, and especially the homes directories. My tape backup unit is a Seagate ST200000A IDE tape drive. I found out the hard way that IDE tape drives are not ideal under linux, and one should use a SCSI unit. I was determined to make this thing work, and finally succeeded.

Introduction

bkp is a set of shell scripts that use 'tar', 'mt', and 'buffer' to perform a backup to an IDE tape drive, although any kernel-recognized tape drive be it SCSI or whatever, should also work. You can also do backups to a file, though you probably wouldn't use bkp for this.

The scripts are designed to do a full backup followed by any number of incremental backups. In my setup, I do a full backup every Sunday night, followed by incremental backups each day of the week. Come Sunday again, the tape is erased and the process starts over again. This is done automatically by calling the scripts as cron jobs, though you can just as easily execute them manually if you like.

I have tested these scripts under Redhat 9 using a Seagate ST200000A tape drive. Any tape drive (ide, scsi, etc.) that you can manipulate with the 'mt' command (/dev/nht0 /dev/nst0, etc.) should work fine. However, I make no warranty on what should or should not work.

Please do not email me on script syntax or "you should have done it this way". I am not a linux guru. I'm sure my scripts can be written more elegantly. All I know is that they work and work reliably. If you have changes and think they can benefit other users, then by all means email me and I'll update things accordingly.

The Files

Following is a description of each file in this package:

bkp.vars

bkp.vars is the main configuration file where you set your system's parameters. YOU SHOULD NOT RUN ANY SCRIPTS UNTIL YOU SET UP THIS FILE. This file is really a shell script that is read by the other scripts. All of the scripts assume this file is located at /etc/bkp/bkp.vars. If you want to keep it someplace else, then you will have to modify the scripts. The following parameters are kept in bkp.vars:

src="/"
The $src variable sets the base or source directory of the backup. Normally, you would set this to "/" which is the root directory of you system. But if you only want to backup certain directories, you can set this accordingly. For example, to backup only the home directory and everything underneath it, use src="/home".

rest="/tmp/restore"
The $rest variable sets the directory where a restore operation will occur. All restored files and directories will be copied to this directory. If you're restoring your entire system, you could set this to "/", although most people will want to restore all files somewhere else, then copy them over, perhaps while in single-user mode. Actually, you don't have to use the restore script (bkprestore.sh)at all, and just use tar for restores.

dev="/dev/nht0"
The $dev variable sets the device or filename for all backups and restores. You always want to use the "non-rewind" version of the device (i.e., use /dev/nst0, NOT /dev/st0).

destdev="1"
Set this variable to "1" if the $dev variable represents a real physical tape device instead of a file. Within the scripts, this allows the 'mt' command to be used accordingly. You should always set this to "1". The only reason why you might want to set this to "0" would be to debug the scripts or do backups to files instead of tape devices. In the latter case, incremental backups to files are not possible; you'll just overwrite your full backup file if you try to do an incremental backup to a file.

blksize="32768"
tarblock="128" # blksize / 512 These two variables set the size of the blocks recorded to tape. A value of 32768 bytes seems to provide maximum performance for "dumb" IDE tape drives, like the Seagate Travan models. The $tarblock parameter must be set to the value of $blksize divided by 512.

exclude="/etc/bkp/bkp.exclude"
The $exclude variable sets the name of a file that contains a list of files and/or directories that you do not want to backup. All entries must be relative to the $src variable. For example, if $src="/", then an entry in bkp.exclude called "proc" would not back up the /proc directory. You never want to backup "proc" anyway, so you should at least include this entry. You may not want to back up /tmp, /dev, or /mnt, and maybe no core files. for this scenario, your bkp.exclude file will look like this:

core
tmp
mnt
proc
dev

Another example: say you don't need to backup joe's home directory, or any mp3 files, your bkp.exclude file will look like this:

core
tmp
mnt
proc
boot
home/joe
*.mp3

The order of the entries is unimportant. Just make sure you do not have a leading slash on any entry, and remember that all the entries are relative to the directory you specified in the $src variable, which for most users will be the root directory.

cmp="-z"
The $cmp variable turns on gzip compression, so the archive is compressed as it is written. Set this variable to "-z" to turn on compression, or empty quotes ("") to turn off. Before you say to yourself, "Gee, of course I want compression" consider the fact that many tape drives have compression built-in, and using gzip will only degrade performance and can actually increase the size of the archive. However, the basic IDE travan style tape drives usually do not have any sort of compression so you would want to set this to "-z".

logdir="/var/log/backup"
The $logdir variable sets the location of the log files. Log files are created when you do full and incremental backups. When you do a new full backup, all of the old log files in this directory are automatically deleted, so you really don't have to worry about setting up log rotation.

datefile="/etc/bkp/bkp.date"
The $datefile variable sets the name of a file that is created or modified every time you do a full or increment backup. The contents are filled with the output of the 'date' command. When subsequent incremental backups are performed, the date file is used to determine which files need to updated from the last backup.

bkpfull.sh

This is the mother of all backup files. When you execute bkpfull.sh, a full backup of every file (minus those specified in bkp.exclude) is performed. This script does the following in this sequence:

1. The tape is rewound.
2. The tape is retensioned.
3. The tape is erased. Yes, all data is destroyed. The first archive will soon be the full backup of all files.
4. All files specified by $src (except those specified by $exclude) are backed up.
5. When the backup is complete, a file with the name <date/time>-full.log will be written to a log file in $logdir.
6. The time of the back up is saved. This is so any subsequent backups will include only the files that have been created or modified since the last full or incremental backup.

bkpincr.sh

This script runs an incremental backup. You would ALWAYS run the 'bkpfull.sh' script first before running this script. Here is what happens:

1. All files are backed up that have a modification or creation date that is later than the previous full or incremental backup.
2. The time of the incremental backup is saved. This is done so additional incremental backups will only include newer or modified files from the last incremental backup.

You can do as many incremental backups as necessary, or until you run out of tape. I would suggest that you limit the number of incremental backups, since each set will have to be restored in sequence during recovery.

bkprestore.sh

This script does a restore. You would manually do a 'rewind' to get to the full backup (always the first archive on the tape), then run this script. You would then run 'bkprestore.sh' over and over again, until there are no more incremental backups left on the tape. The process overwrites older files in the restore directory with the latest versions (and new files) from each incremental backup set to the restore directory. After all incremental backups are done, the restore directory contains the complete and latest set of backed up files.

erase list retension rewind status

---

These scripts are not used by the backup or restore scripts above, and are stand alone utilities. They provide as an easy way to manipulate your tape drive.

erase - Erases the tape. You are prompted whether or not to continue.

        Note that erasing a tape only involves writing two end of file
        marks at the beginning of the tape, so the operation is fairly
        quick on most drives. However, some drives literaly erase the
        whole tape, which can take a very long time. If you have one of
        these types of drives, I suggest you remove the 'mt erase' line
        in bkpfull.sh.

list - lists the current archive to the screen. The tape must be positioned

       at the file or archive that you want to list prior to running this
       script. To list all files on the tape, first run 'rewind', then run
       'list'. You will get a list of files in the full backup. If you run
       list again, you will get a list of the first incremental backup.
       You can keep running 'list' until no more incremental backups are
       present. Note that 'list' does not do any writing to the tape or
       restoring any files, it just shows you what's on the tape.

retension - Retensions the tape. This causes the tape in the cartridge to

       wind foreward to the end of the tape, then back to a full rewind.
       If a tape is not properly tensioned in the cartridge, the tape iteslf
       may lift slightly away from the magnetic head and cause errors.
       It is a good idea to retension the tape after many "jogging" operations,
       that is, when the tape does a lot of reading, writing, or positioning.
       The 'bkpfull.sh' script automatically retensions the tape for you before
       doing a full backup.

rewind - Rewinds the tape to the beginning. The tape is positioned and ready

to read, list, or write the full backup.

status - Reports the tape drive status. What you see on the screen will vary

         depending upon the type of tape drive you have. But virtually all
         tape drives will show you the current block where the tape is
         positioned and the block size in bytes.

Disaster Recovery

Hopefully, you will never have to follow this procedure. In the worst case scenario, your hard drive fails and you can no longer boot your system. Here is the steps you should take for recovery:

1. Boot from your linux distribution CD and run rescue mode. Before trying to restore to tape, you should try and recover the data using this and any other means available to you.
2. If you discover that your hard drive is dead and the data cannot be restored, get a new hard drive. I would strongly recommend that you do not try and reuse the drive, even if it can be reformatted and partitioned successfully again. The failure will likely happen again.
3. Do a fresh install of your linux distribution. Copy the bkp files to the /root directory from your backup floppy (You do keep a copy of the bkp files on a floppy, don't you?).
4. Boot your linux box in single-user mode. Make sure that the bkp.vars file is set correctly. Insert your backup tape and run 'rewind' followed by 'bkprestore.sh'. This will restore the full backup archive. If you don't have access to bkprestore.sh, you should be able to restore from tape using the following tar command:

   /bin/tar $cmp -x -v -v -C /tmp/restore -f /dev/nht0

   Of course, subsitute the restore point (/tmp/restore) and the device (/dev/tape) with your own.

5. When the full backup completes, run 'bkprestoresh' again. This will restore the first incremental backup. When that completes, run it again. Keep doing this until all the incremental backups have been restored. Your tape drive will complain with an error message, or display no output at all when no more archives are available on tape.

Additional Notes

Like most backup programs, bkp will restore deleted files. For example, suppose you made a full backup and then deleted some files permanently from your system. Those files will be restored from the full (or incremental) backup sets. Generally this is not a problem, but you need to be aware that files you have deleted may be resurrected during a restore operation.

Absolute paths are used for 'tar', 'mt', 'sed', 'date', 'head', 'tail', and 'buffer' in the scripts. This is done to avoid problems when running the scripts under cron. If these programs are not located in their usual places, you may have to modify the scripts accordingly. Also, you probably do not have Lee McLoughlin's 'buffer' program, but it has been around a long time and is available from the usual sources, both in source and rpm versions.

When run as cron, the scripts will output the head and tail of the log file to stdout. Under cron, this should send you email with this abbreviated log.

bkp is not set up for multiple tape backups, and the use of the buffer program precludes this. Even if you add the -M flag to the tar commands it won't work. If you decide not to use 'buffer', you will degrade compressed backups by at least a factor of 3 or higher. If you segregate your backups by tape and they fit on one tape, an alternative is to have multiple copies of bkp, one for each tape. Just set up separate bkp.vars and bkp.exclude files to handle the portions of your system that require backups that occupy a full tape.

There are lots of "gotchas" with tape drives, especially older IDE models. Some require you to disable dma (hdparm -d0 /dev/hxx). Others require scsi emulation. If you experience problems with your particular tape drive model, turn to usenet. Most likely, somebody else has already had the same problems you may be experiencing.