# scdbackup-0.8.6
http://scdbackup.webframe.org scdbackup@gmx.net Thomas Schmitt
http://scdbackup.sourceforge.net
- Purpose
- backup large amounts of data on one or more CDs or DVDs, simple (therefore no excuse not to do the backup), no special tool needed for reading the backup
- Formats
- ISO9660 file systems (readable after : mount ) afio archives (readable by : afio ... /dev/cdrom ) star archives (readable by : star ... </dev/cdrom )
Commands for CD:
scdbackup_home ... backup of the user's $HOME directory. ISO9660
scdbackup ........ command for defining own backup collections and for
spontaneous data copies. ISO9660. Example:
scdbackup /home /home2=/usr/home -not /home/guest
Backup /home and /usr/home . But use /home2 on CD as
name for /usr/home and omit subdir /home/guest
scdbackup_afio ... like scdbackup but with afio archives rather than ISO9660.
This may also produce star-Archives.
scdbackup_sys .... backup of system directories as afio archives (or star) scdbackup_verify . test a resulting CD for correct readability scdbackup_blank .. blank a CD-RW to make it writable again scdbackup_askme .. read an info script from the backup media and perform
search operations much faster than the script could do scdbackup_clear .. release all locks (in case one has survived the end
of a script).
For DVD according commands are available. They all begin with "sdvdbackup" : sdvdbackup_home, sdvdbackup, sdvdbackup_afio, sdvdbackup_sys, sdvdbackup_verify, sdvdbackup_blank, sdvdbackup_askme, sdvdbackup_clear
Software requirements (see appendix LINKS for addresses): /bin/sh compatible to GNU Bourne-Again SHell (bash) or Korn Shell (ksh)
including usual programs like: cat, date, du, echo, ls, ...
see hints for working around shell compatibility problems in
script change_shell_to_use .
mkisofs from package "cdrtools" generates ISO9660 file systems which are
accessible with nearly any modern system.
afio or star archiver programs (for system backups)
gzip data compression program (for incremental backup management)
Required for writing CD :
cdrecord from package "cdrtools" for writing CD on Unix style systems
or
cdrskin a limited imitation of above on base of libburn on Linux >=2.4
Required for writing DVD (see also Appendix DVD) : growisofs from package "dvd+rw-tools",
or
cdrecord-ProDVD, a binary with time limited license
This program system is constantly tested on Intel/AMD Linux systems. Ports to other usable systems are appreciated.
Special features:
Breaks the 650 MB / 4.7 GB limits (more than one CD / DVD )
Information script on each CD / DVD tells where a certain file may be found
Collision test with renaming avoids aborting of mkisofs
Multi user safe by use of a locking protocol
Interrupted backups can be resumed at the start of the failed piece
Incremental backups with an arbitrary number of levels
Checksums appended to media
Optional multi copy backups provide ability to repair damaged volumes
Table of Content:
- Planning (of usage and installation)
- Installation (at first for a normal desktop user)
- Superuser Planning (of an installation used by the superuser)
- Usage Backup Commands ISO9660 Filesystems afio Archives (or star archives) Restartability Verifying CDs and DVDs Blanking CD-RW , Erasing data from DVD Incremental Backups Backup Information Scripts Access Permissions (of the scdbackup installation)
- Inner details - Appendice DIRECTORIES Directories of a scdbackup installation CONFIGURATION Configuration files and variables WORKERS Operational components of scdbackup (the software) RECOVERY Precautions for and recovery from disasters AUTOMAT Issues of unattented backup runs VERIFY Format of checksum records, tags, block checksum lists REDUNDANCY Repairable multi copy backups CRYPTO Encrypted afio or star backups ATTRLIST Recording and restoring ACLs and Extended File Attributes LINKS Links to supporting software GPL Fulfilling the GPL of mkisofs_prog and cdrecord_prog DVD How to get and install growisofs resp. cdrecord-ProDVD DVD-TYPES Using DVD+RW, DVD-RW, DVD-RAM IDE Using consumer grade burners with Linux NET Burning via a remote scdbackup installation over network TUNING Options to make scdbackup faster or leaner INODE Standard method of incremental backup and alternatives SNAPSHOT How to deal with filesystem snapshots WRITER Interface specifications for write adapter programs
The following text mostly refers to CD media but is also valid for DVD. If not explicitely outruled, one may safely substitute the CD commands scdbackup* by the DVD commands sdvdbackup* .
Planning
Burning of CD or DVD on Linux relies on applications which format and record the data. The most commonly readable format is ISO-9660. scdbackup's configuration will request some parameters from you. So you have to find out the appropriate answers and eventually prepare resources. The non-trivial parameters and eventual preparations are described in the following text up to paragraph "Installation".
If there occur problems with using the recorder then ask the sysadmin to read paragraph "Permission to use the recorder" and ask for appropriate remedy.
If you got decent cdrtools properly installed then you will get advised to use those. So the configuration question "Use mkisofs_prog and cdrecord_prog ? (y/n/b)" will most likely get input "n" or "b".
If your ISO formatted disks shall be read with long names by common MS-Windows systems then you will have to add "-J" to that "n" or "b" (like "b -J"). Regrettably the capabilities of the Joliet Extension are quite restricted and there are versions of mkisofs which fail with -J on some backup situations. If you experience aborts of mkisofs with "-J", try to add "-joliet-long". But then you have to check readability with the particular MS-Windows system. Without -J, files will be visible with shortened DOS-style names and bear their proper content, anyway.
When configuring for CD you will have to enter a "SCSI-Address of CD recorder". If in that moment you do not see a list of addresses like
------------------- SCSI devices. To be used like 0,0,0
0,0,0 0) 'HL-DT-ST' 'DVDRAM GSA-4082B' 'A201' Removable CD-ROM
1,0,0 100) '' '' '' Removable not present Disk
------------------- end of SCSI device list
------------------- ATA devices. To be used like ATA:0,0,0
------------------- end of ATA device list
then check wether cdrtools are installed on your system. There is a Linux
x86-binary cdrecord_prog included with scdbackup but you will need the
superuser to get it operational.
If you see the list, then it should be possible to identify the line with
your recorder's name and its SCSI address which here is "0,0,0".
If the burner had been listed in the section for "ATA devices" then its number
code would need to be prefixed by "ATA:". Like "ATA:0,0,0".
The usable addresses have undergone some evolution following the changes in hardware customs. Real SCSI CD recorders have become rare. Nowadays IDE (actually ATAPI) hardware is built into consumer PCs. For a while, the kernel module ide-scsi made IDE recorders appear like real SCSI devices. The following table tries to give you an overview over Linux kernels and possible addresses for CD burners :
Kernel real SCSI IDE + ide-scsi IDE (Joerg) IDE (Linus) 2.2 0,0,0 0,0,0 -unknown- -not usable- 2.4 0,0,0 0,0,0 ATA:0,0,0 -not usable- 2.6 0,0,0 -deprecated- ATA:0,0,0 /dev/hdc
The Joerg-style addresses know other prefixes like "REMOTE:". Run
cdrecord dev=HELP
to get a list. Read the prefix from lines "Transp. layer ind.:".
Use prefix without ":" to learn about the according devices. Like :
cdrecord dev=ATA -scanbus
Note: ATA does not work while the recorder is guarded by ide-scsi.
With question "Speed ?" you should enter the maximum value for rewritable media. Slower media will be handled properly by cdrecord.
Configuration for DVD usually demands an operational command growisofs . Best is to try one of the examples on its man page wether it works for you in the role of the current user. You will need to know the DVD writer's device file address. /dev/dvd may or may not be right. If your sysadmin doesn't know (after reading Appendix DVD) and your system management software doesn't tell, then close the drive trays and try commands like eject /dev/sr0 or eject /dev/dvd . If it reacts, then that address is worth a try.
You may also use program cdrecord-ProDVD instead (see appendix DVD). It uses addresses as described above for CD.
"Speed ?" will be "any" for good media and "1" for poorly performing disks. For cdrecord-ProDVD numbers like "2" or "4" are appropriate.
You will get asked for a "Large file split directory" and a "Buffer file name". You will only need that if you got single files larger than approximately 600 MB (or 2 GB minus 8 kB with DVD) or if your computer system and burner are too slow and too picky to avoid misburns due to lowspeed data transfer.
If you do not expect the need for said "split directory" or "Buffer file" then skip now to "Installation".
Disk buffer
There may be two buffer objects which each can occupy as much space on your hard disk as fits on a single media (i.e. 650 MB resp. 4.7 GB). Therefore it is possible that you cannot afford both or even a single one of them.
The split directory is needed with DVD for files of size near 2 GB or larger. That is because a lot of reader software does handle files properly only if they are a bit smaller than 2 GB. With CD you will need a split directory if you want to backup files that do not fit on a single media. If there are such oversized files and if there is no split directory given then those files are excluded from backups, unless this security feature has been disabled explicitely.
The buffer file is eventually used for assembling the data of a backup volume on disk before they are handed over to the burner without putting a high load on the system. This may be necessary if the system is heavily loaded or if a compressed archive format is to be written and if the burner hardware got no protection against buffer underruns (Burnproof, Smartburn, etc.).
If you got oversized files and want to backup them, you will absolutely need a split directory. The buffer file should not be necessary if you got modern hardware. (It is useful for advanced purposes, nevertheless.)
After you decided to have a split directory, find a suitable filesystem and
create a new empty directory there. For a buffer file you would just have to
choose a name in an existing directory on a suitable filesystem. Suitable
here means : with enough free space.
You must have separate directories resp. files for CD and DVD unless you
only got one single burner which is in charge for both media types. I strongly
recommend to use different names in any case.
Set the access permissions of the newly created split directories to rwx for the interested user(s). (In most frigthening case: chmod a+rwx ) Make sure the directory of an eventual bufferfile offers rwx-permissions to all those users.
Each single scdbackup installation (i.e. for security reasons : each user) should use own addresses for their buffer directories and buffer files.
Installation
To achieve a clear security situation each user should have an own separate scdbackup installation.
Steps 1 and 2 must be done by the particular user who shall use that installation. This is not necessarily the superuser but rather the owner of the backup data.
The superuser may follow these steps for the installation of an own, very private backup system. But superuser authority will elsewise only be needed for a few optional activities. If programs cdrecord and/or growisofs are installed properly on the system and may be used by the installing user for burning, then there is no urgent need for superuser actions at all.
Each of the steps may be repeated to restore original software and permission settings or to edit the configuration.
--- Step 1 :
Copy the archive to a directory where you can create a sub directory scdbackup-0.8.6, go there, unpack the archive and go to the sub directory inst newly created by this. For example use $HOME/scdbackup :
mkdir $HOME/scdbackup
cd $HOME/scdbackup
tar xzf $HOME/downloaded_stuff/scdbackup-0.8.6.tar.gz
cd scdbackup-0.8.6/inst
If the superuser is doing this, it is also absolutely necessary to do
chown root ../../scdbackup-0.8.6/*/* ../../scdbackup-0.8.6/* \
../../scdbackup-0.8.6
and best to issue an appropriate chgrp command, too.
--- Step 1b (only needed when upgrading from an older version) :
If you already had installed another version of scdbackup, you may now copy some history data and the old configuration as a template into the new directory :
./FETCH_CONFIGURATION
This refuses to overwrite any files so it only makes sense in a freshly unpacked directory. Any users afterwards are still configured to use their old installation.
If you maintain incremental backup configurations and suffered from long planning times you may be interested in reading the first paragraph of appendix INODE below.
Only if you programmed own applications which make use of inner parts of
scdbackup previous to version 0.8.5, only then you should now read appendix
DIRECTORIES.
The mere use of commands scdbackup* or sdvdbackup* within your programs
will continue to work without ./COMPATIBILITY_LINKS, of course.
Unless the user id number is 0 the following scripts write the files .scdbackup_dir and .scdbackup_lang into the user's $HOME-Directory. They eventually write file .scdbackup_userid into the user's $HOME-Directory.
--- Step 2a if you got a CD writer (for DVD writer only, see Step 2b below) :
While the CD-Recorder is not busy, run :
./CONFIGURE_CD
Several questions have to be answered:
SCSI-Address of CD recorder ? (e.g. 3,0)
Speed ? (e.g. 2)
Automatically erase CD before writing ? (y/n)
Use mkisofs_prog and cdrecord_prog ? (y/n/b)
Large file split directory ? ( "-" = exclude large files)
Buffer file name ? ( or "-" or "-afio_compress" )
Estimated capacity for a compressed afio CD ? (eg. 1024m)
Exclusion list of directories for system backup ?
Apply locking protocol ? (/path yj1 n0)
( With other shells than bash : Your user id number ? )
When asked about mkisofs_prog and cdrecord_prog enter the letter given after "Advised input is :" unless you got good reason not to do so. (Let me know about it)
If no disk buffer (see above) is needed then answer the question "Large file split directory" by a single "-". "Buffer file name" should be "-afio_compress" for true burnproof hardware and media, else enter "-" or a filename.
The "Exclusion list of directories for system backup" is only of interest if you are superuser right now. As normal user just press the Enter key.
To "Apply locking protocol" answer "n" unless you read "Superuser Planning".
--- Step 2b if you got a DVD writer (works with or without Step 2a) :
If command growisofs is not found by your shell you will have to install dvd+rw-tools as described in Appendix DVD.
While the DVD writer is not busy, run :
./CONFIGURE_DVD
Several questions have to be answered:
Device file address of DVD writer ? (e.g. /dev/dvd)
Speed ? (e.g. any , 1)
Automatically erase DVD before writing ? (y/n)
Use mkisofs_prog ? (yj1/n0)
Large file split directory ? ( "-" = exclude large files)
Buffer file name ? ( or "-" or "-afio_compress" )
Estimated capacity for a compressed afio DVD ? (e.g. 8000m)
Exclusion list of directories for system backup ?
Apply locking protocol ? (/path yj1 n0)
( Non-bash users also may be prompted for a unique user id number )
When asked about mkisofs_prog enter the letter given after "Advised input is :" unless you got good reason not to do so. (Let me know about it)
If no disk buffer (see above) is needed then answer the question "Large file split directory" by a single "-". "Buffer file name" should be "-afio_compress" for true burnproof hardware and media, else enter "-" or a filename.
The "Exclusion list of directories for system backup" is only of interest if you are superuser right now. As normal user just press the Enter key.
To "Apply locking protocol" answer "n" unless you read "Superuser Planning".
--- Step 2c. (Only with private installation for superuser)
For his own private installation of scdbackup, the superuser has to run :
./ADD_USER
Make no mistake : only if you are already superuser now and only with the installation which shall do the superuser's backup jobs and where no other user is permitted.
After Configuration
Permission settings after a first configuration will not allow others to use your scdbackup installation. This is for your security. Stay with it.
The commands of scdbackup are now accessible in subdirectory ./cmd of $HOME/scdbackup/scdbackup-0.8.6 . Make a first test :
$HOME/scdbackup/scdbackup-0.8.6/cmd/scdbackup_home -version
should reply "scdbackup 0.8.6 2005.02.08.173050" (timestamp varies).
You may now add $HOME/scdbackup/scdbackup-0.8.6/cmd to your PATH variable :
export PATH="$PATH":"$HOME/scdbackup/scdbackup-0.8.6/cmd"
and then execute commands like
scdbackup_home -version
There are startup scripts like .profile or .bashrc where you may add this directory path permanently. Ask your sysadmin.
If this is an upgrade from scdbackup-0.8 or older then the superuser will have to remove the old commands in /usr/bin (or wherever they are) or to overwrite them as described in the next paragraph.
If this is your first encounter with scdbackup then i advise you to next
read "Usage" or just try
$HOME/scdbackup/scdbackup-0.8.6/cmd/scdbackup_home
(without "-version").
The eventual superuser activities can wait for an hour - if cdrecord and
DVD writing device are in proper state, that is.
Superuser Part of Installation
The superuser may decide to copy the scdbackup commands to a public command directory rather than to teach his users how to permanently set their PATH. These command scripts are quite small and merely direct the users to their own scdbackup installations. Their only hazard is to get tricked into a fake scdbackup installation by the user's own environment variables SCDBACKUP_DIR or HOME or by the user's own configuration file $HOME/.scdbackup_dir . Regular installations of other users should repel such an attempt, anyway.
With upgrades from scdbackup-0.8 or older, the superuser has to make sure that the old installed commands get removed or overwritten by the new ones. Commands "which scdbackup" and "which scdbackup_sys" should tell where they are located.
Permission to use the recorder:
Under normal circumstances there should be a system installation of cdrtools with sufficiently open permissions. The user then was advised to enter "n" or "b" during CD configuration.
In the case of system installed cdrecord, of course, a user needs permission to use it with setuid root in order to burn CD. (Strongly deprecated: via sudo) Similar is true for growisofs (+x) and its DVD device file (+rw) with DVD. (sudo is strongly deprecated for growisofs)
Nevertheless, if the user decided to use Linux binary cdrecord_prog, then the superuser has to take it into his ownership and to set its setuid bit.
A mistrusting superuser may decide to perform those tasks manually following this template (copy commands to /usr/local/bin and /usr/sbin ):
target=/usr/local/bin
chmod a-w ../scdbackup-0.8.6/cmd/
cp ../scdbackup-0.8.6/cmd/ "$target"
chown root "$target"/scdbackup* "$target"/sdvdbackup*
chmod a+rx,u+w,go-w "$target"/scdbackup* "$target"/sdvdbackup*
chmod go-x "$target"/scdbackup_sys "$target"/sdvdbackup_sys
(have a look into /usr/local/sbin wether there is s*dbackup_sys to be removed)
resp.
chown root ../scdbackup-0.8.6/bin/cdrecord_prog chmod a-w,a+rx,u+s ../scdbackup-0.8.6/bin/cdrecord_prog
But there is a script which does this more comfortably, reliably and avoids
any "*" in the commands.
If the superuser intends to use scdbackup for own backup runs, then there is
no reason to particularly mistrust the following script. (Bug reports welcome)
--- Step 3 (optional, see possible reasons above)
Become superuser now and run (within directory scdbackup-0.8.6/inst)
./SUPERUSER_CONFIGURE
This script has two purposes:
- If CD writing has been configured in Step 2a it makes the superuser owner of
cdrecord_prog, sets its setuid bit and grants to everyone the permission to
execute the program. This is necessary if the user decided to use
mkisofs_prog and cdrecord_prog rather than the locally installed versions
of mkisofs and cdrecord.
- It copies the command scripts scdbackup* to a public command directory like
/usr/bin or /usr/local/bin if CD writing is actually configured. If DVD writing has been configured, then the commands sdvdbackup* are copied to that directory.
Both tasks are performed separately and controled by two user inputs: Do you want to enable access to the CD recorder via cdrecord_prog ? (y/n) Install directory ? (e.g. /usr/bin , "-" = do not install)
The user should afterwards check wether
scdbackup_home -version
yields the same result as in the first test :
$HOME/scdbackup/scdbackup-0.8.6/cmd/scdbackup_home -version
If the programs mkisofs_prog and cdrecord_prog are not to be used, but rather the system's programs mksiofs and cdreord, then there is no need to repeat step 3 with further user's installations.
If the superuser does not have an own scdbackup installation yet, then this
would nearly be the time to start -as superuser- with step 1 in order
to get one.
But first read this :
Superuser Planning
The superuser will most likely use format afio unless there are user data to take care of, which clearly benefit from ISO 9660 format. By help of script star_as_afio_wrapper one may alternatively employ the archive format of program star. (See example in appendix WORKERS.)
The idea is to have a rather slim system backup, which is made with
scdbackup_sys
Be aware that scdbackup_sys does not create a recovery system but just records the files (even quite special ones) in an afio archive. This conserves the system configuration, but for re-installing you need to do a base installation of Linux first. The base installation has to be compatible with the system you made the archive from. So make copies of your Linux CDs to have a compatible system at hand when needed.
You may also use program star as intended by its author to make a full backup of the system partitions. This will not need copies of the originally installed Linux distribution media.
See appendix RECOVERY for details of disaster precautions and recovery.
Alternatively you may choose to use a disaster recovery system like mkCDrec ( http://mkcdrec.ota.be/project/index.html ) which tries to save you as quick as possible from the consequences of big accidents.
Nevertheless, if you cannot get the old hardware running again it might be
impossible to use system backups of any kind flatly. Then you have to install
a completely new operating system and may use the old system backup only for
as source of configuration files.
I myself never lost a complete system but had occasions when an older
reference copy of the system files came in very handy.
User data which rely on exact file attributes should not be stored as ISO9660. For such data, one should better use scdbackup_afio .
Usually there are several kinds of data on a Linux disk, which should be
backuped:
Persistent system objects, often with special attributes and permissions
Installed application packages (often below /opt) User created or collected data (often below /home) Other kinds of data do not need backup or would even make problems : Volatile system objects (e.g. /proc) Temporary files (e.g. /tmp or $HOME/.netscape/cache) Removable filesystems (/mnt /floppy /cdrom)
During installation, you will be asked for "Exclusion list of directories
for system backup".
This list should contain all directories under / , which do not contain
persistent system data.
Example (have a look in your own / directory for more candidates) :
/opt /home /mnt /floppy /cdrom /cdrw /tmp /proc /sys
All other subdirectories of the root directory will be covered by the system backup. Among the excluded directories, only /opt and /home need backup.
If there is only one real user who has not got a giant amount of data, a combined backup of /opt and /home is appropriate. Since my Netscape cache is quite large, i exclude it.
scdbackup_afio /home /opt -not /home/thomas/.netscape/cache
If there are very large amounts of data, it will be convenient to have separate backups. These backups can be run with different frequency. /opt will not need to be backuped as often as /home/projects .
scdbackup_afio /opt
scdbackup_afio /home -not /home/projects /home/thomas/.netscape/cache
scdbackup /home/projects
Real users (like "thomas", "guest1", "guest2") should do separate backups with
scdbackup_home
or
scdbackup_afio $HOME
Exclude their $HOME directories from the general data backup
scdbackup_afio /home \
-not /home/thomas /home/guest1 /home/guest2
Next you should make a time schedule, when to do which backup. It is generally a good idea to have at least three older backups. From time to time a permanent backup could be made on CD-R in order to be kept forever.
On the other hand ... be aware what traces you leave for future archeologists.
With several concurrent real users you will possibly have to coordinate them
by scdbackup's locking mechanism in a common directory. Consider to give
/tmp
as answer to configuration question "Apply locking protocol".
Then scdbackup will try to prevent concurrent access to unique resources.
Permission settings after an installation will not allow others to use a scdbackup installation, unless these restrictions are lifted by ./SET_PERMISSIONS (see below "Access Permissions") or unless laxer settings have been fetched during ./FETCH_CONFIGURATION. Interested and authorized users then go to the installation directory and perform ./ADD_USER .
Now go and -as superuser- begin with step 1 of the installation procedure. If the superuser did not use scdbackup before , execute as step 1b :
./SUPERUSER_FETCH_CONF /home/thomas/scdbackup/scdbackup-0.8.6
to get the previously made settings of user "thomas" as templates for own ./CONFIGURE_CD resp ./CONFIGURE_DVD . Else
./FETCH_CONFIGURATION
is ok.
In any case : check what you confirm with ./CONFIGURE_CD and ./CONFIGURE_DVD .
Take care not to allow others to access your eventual disk buffer objects. Best is to have none, if one can live without them.
Usage
scdbackup offers different formats for the data on media.
The data format preferable for user data files is in most cases ISO 9660. The media are mountable and the files may be used directly by any program that can handle files which offer no write access. This format is also most likely to be accessible on other operating systems.
In the most simple case, command scdbackup_home is sufficient. For arbitrary data locations use command scdbackup .
As said, this is ok for directories trees and data files of a single
user, where write permissions are general and simple.
To reliably cope with symbolic links you will need mkisofs from
cdrtools 2.0 or higher.
With old mkisofs_prog included in scdbackup's tarball files with the character
"=" in their name may have to be stored in the backup with a different name.
scdbackup will eventually replace "=" by "_". This may happen if with
./CONFIGURE_CD the answer to question "Use mkisofs_prog and cdrecord_prog ?"
was "y". With a modern mkisofs you will be advised to enter "n" or "b".
Regrettably ISO 9660 and the formatting program mkisofs have certain
weaknesses when it comes to special (or even exotic) file types, including
very wide or deep directory trees. Additionaly the filesystem driver
community is not very trustworthy in respect to w-permissions on
read-only disks. Be prepared to restore them manually after having copied
the files back from CD to hard disk.
Another disadvantage is that scdbackup does not offer data compression
together with ISO format.
For such purposes, scdbackup offers the format of the reliable archiver afio or the format of Joerg Schilling's backup program star. The commands scdbackup_afio and scdbackup_sys produce these formats.
Backup Commands
The backup creating command scripts of the scdbackup-system all accept
the options described by
cd_backup_planer -help | less
but usually it is only necessary to provide some directory or file names.
See also examples.html for commented usage examples.
ISO9660 Filesystems
These filesystems can be mounted on devices like /dev/cdrom or /dev/dvd . They also can be exchanged with other systems like Windows or Mac.
scdbackup_home a command that backups the user's $HOME directory on CD
except the Netscape cache. No arguments needed.
sdvdbackup_home DVD version of above.
scdbackup a command that backups the files and directories
given as arguments by the user.
These objects may be renamed by giving the target name
before '=' and the source name.
Example: rename /usr/home (on disk) to /home2 (on CD)
/home2=/usr/home
Subdirecories and files may be excluded by writing them
behind option -not. Example: exclude three directories
-not /home/guest /home/kindergarden /usr/home/guest
Complete example:
scdbackup /home /home2=/usr/home -not /home/guest
Backup /home and /usr/home . But use /home2 on CD as
name for /usr/home and omit subdir /home/guest
sdvdbackup DVD version of above.
Directory access permissions may be restored by the compressed shell script
/added_by_scdbackup/DIR_PERMS.sh.gz
on the last volume of such a backup. (See also examples.html#restore_incr )
This may be necessary because directory access permissions on a mounted CD
may not be the same as they were during the backup.
afio Archives
afio archives can be read with the CD-ROM or DVD drive while the media is
inserted but not mounted.
Check Archive:
afio -tv /dev/cdrom
Re-install (beware, only if you need):
cd /
afio -ivZ /dev/cdrom
Example : Extract X11 configuration files to a harmless place
No need to be superuser for that !
mkdir $HOME/system_reference_copy
cd $HOME/system_reference_copy
afio -ivZ -y 'etc/X11/XF86Config*' /dev/cdrom
Now you may compare them with those of your current system
diff etc/X11/XF86Config /etc/X11/XF86Config | less
Read for details : man afio .
By help of script star_as_afio_wrapper one may alternatively create archives in the format of program star, provided that program is installed. For an example see appendix WORKERS.
scdbackup_afio like scdbackup but with afio archives rather than
ISO9660 file systems. If there is a buffer file or
if compression was set during configuration then
the archives contain compressed files.
scdbackup_afio /home -not /home/guest
sdvdbackup_afio DVD version of above.
scdbackup_sys a command that backups the root directory as afio
archives. Some well known non-system directories and
some trash directories are excluded. (My apologies
to all those who work hard on programs that fill in
the "trash" :) Arguments can be given to include or
exclude other directories.
The exclusion list is set by CONFIGURE . Example:
/cdrom /cdrw /floppy /home /mnt /opt /proc /tmp
Example (to be executed as superuser):
include /opt (which usually is excluded)
but omit /home2 and /usr/trash
scdbackup_sys /opt -not /home2 /usr/trash
sdvdbackup_sys DVD version of above.
If the size of the resulting afio archives exceeds 650 MB then they are split into pieces of 650 MB and stored on several CDs which form a multi volume archive. Such archives have to be read from their first CD up to the one which contains the desired data. If a CD is missing then all following ones may be unreadable. DVD split size is 4480 MB.
The main advantage of compressed multi volume archives is their most effective use of CD space. Single volume compressed archives usually need quite a lot of spare space.
The ability to create multi volume archives allows you to risc larger single volume ones. Worst case is an additional CD with a few MB on it. (You may adjust the input size by -max_size number )
It is convenient to have a copy of the program bin/raedchen (e.g. on floppy)
to be able to concatenate the parts of such an archive.
Check:
raedchen -quiet -pipe /dev/cdrom 650m "cat" \
-post_pipe_command "eject /dev/cdrom" | afio -tv -
Restore:
raedchen -quiet -pipe /dev/cdrom 650m "cat" \
-post_pipe_command "eject /dev/cdrom" | afio -ivZ - This is necessary because the CD returns some more bytes than have been written. The false extra bytes spoil at least the file which is stored at the end of the first and the beginning of the second CD.
With DVD use :
raedchen -quiet -pipe /dev/dvd 4480m "cat" \
-post_pipe_command "eject /dev/dvd" | afio ...
If you are completely without scdbackup, then you will have to use a shell
loop for concatenation. Like this ( 332800 = 650 MB / 2 kB ) :
( while test -n 1
do
dd if=/dev/cdrom bs=2048 count=332800 || exit 0
eject /dev/cdrom
echo 'Next media please' >&2
read dummy
done ) | afio ...
There is an appropriate option to make the verify commands handle such overflown media. (see below)
The actual CD overflow size for afio is set by configuration file scdbackup_media_cap_value or shell variable SCDBACKUP_MEDIA_CAP . For DVD it is set in sdvdbackup_media_cap_value resp. SDVDBACKUP_MEDIA_CAP. If you change the defaults 650m / 4480m take care to record the new size on paper or on the raedchen floppy.
Restartability
All the above backup commands can be executed with first argument -resume and
an optional number as second argument. They then restart an existing backup
script at the backup piece with the given number. A dash (-) preceding the
number will cause only this particular volume to be created and the script to
end afterwards.
If the number is missing (or the second argument is "AUTO") then the script
resumes at the piece depicted by the status file ({skriptname}.pieceno).
scdbackup , scdbackup_sys and scdbackup_afio additionaly recognize these options if one of them is given as first argument : -prepare_only prepares for -resume but does not create CDs -last_volume_count prints number of volumes of most recent backup See script outer_loop for an example how to use them.
Verifying CDs and DVDs
First of all : the verification command described here does not ensure that the backup contains what you want to backup. Wether you and scdbackup did it right has to be proven by looking at the meaning of the content i.e. wether the byte size is plausible and wether expected filenames are on CD (locatable by ASKME).
This verification nevertheless ensures that the data stream has been transfered to the CD successfully and can be read without alteration. (There is only a very very remote chance of 1 to 3e38 that an alteration will be unnoticed.) Therefore it checks the CD recorder, the CD media and the CD reading device.
In order to be verificable, a CD must have been created by version 0.8 -or
more modern- of scdbackup with standard content of file
scdbackup_make_checksum_value .
With these settings the commands scdbackup, scdbackup_home, scdbackup_sys and
scdbackup_afio record the length and a 128 Bit MD5 checksum of every backup
volume that is written. This record is stored in a recordfile on your disk
and may be used later to checkread the CD.
Beginning with version 0.8.3 the record is also stored as a checksum tag
at the end of the volume itself.
scdbackup_verify Reads raw data from a CD and checks wether these are
the same as the ones handed over to cdrecord. It reads
CD data from /dev/cdrom and compares them with the most
recent volume record.
scdbackup_verify -auto
tries to gather records from any source and checks wether
one of them matches or obviously indicates a mismatch.
Other data sources (CD devices) or older records may be
used by options to scdbackup_verify. E.g.:
scdbackup_verify /dev/cdrom -auto
For backups made with scdbackup 0.8 to 0.8.2 you need to
keep the record list. Backup it.
See output of scdbackup_verify -help resp. file
scdbackup_verify_help for a description of options,
the recordfile's name and more examples.
The command reads the whole CD and reports its result by
a text on stderr as well as by its exit value:
* "OK ..." and an exit value of 0 if the verification
process was successful.
* Various complaints in CAPITAL LETTERS and exit value 1
if verification fails (i.e. damaged or confused CD).
sdvdbackup_verify DVD version of above. Uses a different recordfile !
The record list usually is stored in file logs/backup_log resp. logs/backup_dvd_log below scdbackup's installation directory. With normal installation settings you can only read that list if you are the same user as the one who wrote the backup.
The list's address may be permanently changed in files scdbackup_checksum_list_value resp. sdvdbackup_checksum_list_value and temporarily overriden by exporting shell variable SCDBACKUP_CHECKSUM_LIST resp. SDVDBACKUP_CHECKSUM_LIST .
Checksum computation and checksum verification are based on cd_backup_planer options -filter_md5 , -compare_checksum and -search_md5 . The format of checksum records and tags is described in appendix VERIFY below.
For multi volume afio archives (see above) you will have to give
scdbackup_verify the media capacity as it was set when the backup was
written. Like :
scdbackup_verify /dev/cdrom -auto - - 650m
sdvdbackup_verify /dev/dvd -auto - - 4480m
If verification fails directly after recording then you may redo the affected volume by a -resume run with the appropriate volume number. Like sdvdbackup -resume 3
One should verify stored backups in regular intervals to get aware of damage
or deterioration.
sdvdbackup_verify -auto
If verification fails with such a check then you should consider to replace
all volumes of the whole backup by a new backup.
If this is not possible then you may make several tries to copy the media
image to a reliable disk and check it there in the hope that the copy attempt
delivered the correct image by incident (this does happen with small damages).
dd if=/cdrom of=/tmp/hopefully_sound_image_file bs=2048
sdvdbackup_verify /tmp/hopefully_sound_image_file -auto
If successful, copy the image to new media
cat /tmp/hopefully_sound_image_file | sdvdbackup -pipe_to_media
If you expect your backups to be not reproducible during the intended storage period (i.e. if they are an outsourced data archive rather than a backup for a life disk), then you should consider to record them with block checksum lists and as several identical copies. See appendix REDUNDANCY.
Blanking CD-RW , Erasing data from DVD
If a CD-RW has been used and shall be re-used, it is necessary to blank it. This process damages the data on the CD-RW . It may be harmful to perform it with CD-R discs. Blanking can be done automatically before writing but depending on the type of recorder it lasts some time. So it is convenient to have some blank CD-RW ready for writing. Also one might sometimes want to apply unusual blanking options.
With DVD+RW there is no blanking required for making media writeable again. Nevertheless, the "shred" option can be very helpful to ensure your privacy when reusing media.
DVD-RW media for use with program growisofs should be blanked "force" once to bring them into a mode with the misleading name "Restricted Overwrite". After that, growisofs will behave with them like with DVD+RW. DVD-RW for use with cdrecord-ProDVD need to be blanked before re-use just like CD-RW. The same is true for growisofs DVD-RW media which are unused or have been formatted to mode "Sequential".
"shred" is a blanking option which does not necessarily make media re-usable. It rather causes the media to be filled with random data. After that, the previously stored data are unreadable by common means.
scdbackup_blank asks you to insert the CD-RW and press 'Enter'.
If an argument is given, this is used as blanking option.
Usual options are "fast" or "all". See output of
cdrecord blank=help
(resp.: cdrecord_prog blank=help) for possible options.
Alternatively, there is option "shred" (see above).
Without argument blanking is set to the option in file
scdbackup_blanken_value. "1" or "y" mean option "fast".
If no other clue is found option "fast" will be used.
Note: If a blanking directive blank=... is part of file
scdbackup_cdrecord_value then no other blanking option
will work.
sdvdbackup_blank DVD version of above.
Its behavior depends on the media type and on the writer
program (growisofs vs. cdrecord-ProDVD).
For DVD+RW only option "shred" does anything senseful.
For cdrecord-ProDVD and DVD-RW : see scdbackup_blank
For growisofs and DVD-RW : Options "all", "full" and
"sequential" lead to mode "Sequential". All other options
lead to the convenient mode "Restricted Overwrite".
Both commands try to find out the type of the media in the drive. This depends much on output of programs dvd+rw-mediainfo resp. cdrecord-ProDVD which is not explicitely mentioned in their documentation. If this classification fails to yield a usable media type, the user is asked to confirm one of the types "CD-RW", "DVD-RW", "DVD+RW". The classification attempt as well as the eventual user inquiry can be avoided by the third argument to the program. Like : sdvdbackup all - DVD-RW
CAUTION : Do not use these commands with one-time writable media ! E.g. if a DVD-R is not recognized properly or declared by the user to be a DVD+RW, then option "shred" will use it up.
After a small backup on DVD it is easy to read old remains from larger backups until they really get overwritten. Therefore before re-use of confidential media apply sdvdbackup_blank shred . Do it multiple times if privacy is a big issue.
The "shred" option tells what will happen next and require another Enter
keypress as confirmation. Then it eventually applies appropriate blanking.
It writes the amount of random data given by the second argument to the
program. E.g.:
sdvdbackup_blank shred 4.7e9
If that second argument is missing then the configuration settings in file
scdbackup_media_cap_value resp. file sdvdbackup_media_cap_value apply.
Usually the preset size of 4480m bytes should be sufficient to hit all databytes on a DVD. Nevertheless you may derive a size from the maximum size given by growisofs during write. Subtract 200 (e.g.: shred 4700372792 ). Remains of backups made with version 0.8.3 or newer will get reported by command scdbackup_verify -scan_print . Like :
- 1_1 A40907.115444 4699824167 7ec3be0efef3f27dd7ec90216d2ee3b4 which tells that backup data may be stored up to byte 4699824167.
Blanking option "shred" produces a checksum record with name "shredded" and therefore together with the appropriate s*backup_verify command is suitable as a quality check for the media.
A large organisation, which presumes to have many enemies, prescribes
additional runs with a character and its complement. It also demands
verification of the random write result. The said organisation does not allow
any erasing of "TOP SECRET" media but demands their complete physical
destruction.
Although i am in doubt wether the complement achieves the desired effect with
CD/DVD media, there are two blanking options "character" and "complement"
which perform the prescribed runs. A compliant procedure would be :
export SCDBACKUP_BLANK_CHARACTER="U"
sdvdbackup_blank character 4.7e9
sdvdbackup_blank complement 4.7e9
sdvdbackup_blank shred 4.7e9
sdvdbackup_verify
The character written is a capital "U" (= 01010101). By variable
SCDBACKUP_BLANK_CHARACTER a different normal letter or digit may be chosen.
Better do not use any special characters.
Incremental Backups
With larger amounts of data it becomes annoying or even impossible to backup the old data at each new run. To avoid this (within certain limitations) one may use an incremental backup scheme.
Such a backup consists of a base backup which might be quite old but contains a full snapshot of all files without regarding their age.
On top of the base backup there are update levels which provide more and more short-term updates. This saves a lot of time and CDs but provides similar security as a set of full backups. Invest some of the savings into increased backup frequency and also keep outdated backup CDs a bit longer. This is to compensate the media redundancy which is reduced with incremental backups.
It has to be emphasized that correct restoring of a multi-level incremental backup is much more complicated than with a full backup. Consider your potential future situation, when you might really need the full content of your backup. Weigh wether avoiding the repeated workload of a full backup really justifies the additional workload in the future crisis situation. Begin with incremental backups if you have to leave out full backups because you cannot afford them.
Initially a backup configuration needs to be created. It records
all parameters of the backup, keeps track of the times and states
of the backup area when backups have been made.
Also the creation process starts the initial base backup which is
called -level 0 .
Choose a configuration name that can be used to create a new directory.
You will need some disk space there for the state records (usually
below 1% of the backup area size).
Let us assume you choose $HOME/my_backup :
$ scdbackup ...the usual arguments for your backup ... \ -conf_dir $HOME/my_backup -level -create_configuration
After that creation, only the name of the configuration and the desired update level have to be given with the scdbackup command. Next day one could execute this command to get a backup of the files that are new or changed since the -level 0 backup :
$ scdbackup -conf_dir $HOME/my_backup -level 1
Options -conf_dir and -level are explained in output of cd_backup_planer -help . There also is an example of a scheme.
Several independant configurations may be defined. Distinct configurations do not interfere even if they share data sources. Not only scdbackup but also scdbackup_home and scdbackup_afio may be used for defining and running incremental backups.
The state records in the configuration directory consume some disk space although they get compressed by gzip. You should watch the size of file $HOME/my_backup/level_0/content_file_list.gz during the -level 0 run and abort it if you cannot afford the space needed. While watching the size do not get worried by the output delay of the gzip pipe. Any substantial size change will get visible in time.
If you cannot afford the disk space, you may disable state recording by using option -disable_safe_leveling together with -level -create_configuration. But be aware that file movements and deletions cannot be detected that way and will therefore not be included in the higher backup levels. Such a backup is still a good source for recovering from limited data accidents but it may be unable to accurately reconstruct the most recent state of your backup area.
The default settings of scdbackup configure a method which depends on the traditional behavior of inode numbers (POSIX "file serial numbers"). If your filesystem turns out to be non-compliant with tradition then you will get suspiciously large backups at higher levels. Please inform me about the type of the affected filesystem and refer to appendix INODE for usage of the old checksum based method.
If you ever need to re-install an incremental backup with multiple levels
you will begin with copying level 0 first (e.g. by use of cp -dR ).
Then you will remove the files which had vanished when level 1 was made.
Then you will copy level 1.
This is repeated with all higher level backups.
For the removal job, there is a compressed shell script
/added_by_scdbackup/VANISHED.sh.gz
on the last volume of each backup level. Don't run it for fun !
(It is interesting to have a look at its content, nevertheless.)
See examples.html for an example restore procedure.
More detail information about state recording can be found in the output of cd_backup_planer -help . Look for "re-install an incremental backup" and also refer to the program options mentioned there.
The decision whether to put a file object into an incremental backup depends on the following properties : mtime (display by ls -l), ctime (by ls -lc), eventually file content checksum (if enabled and an ancestor directory has young time stamps), eventually the inode and decice numbers (if enabled), an economy consideration which eventually includes whole directories if there are many small files to be included.
Backup Information Scripts
Each backup CD contains a shell script named ASKME which can tell on which one
a given fileaddress may have ended up in the backup. Since it only knows the
fileaddresses given to mkisofs, this script cannot always tell for sure
wether a file was actually present at the time of the backup. But it can
tell the CD where one has to look.
In case of incremental backups the information scripts of all older backup
levels are also present on each CD and can be used for searching.
See text askme_help for usage information.
With ISO9660 formatted backups, the script is located in the root directory
of the backup volume.
With archive formatted backups (afio, star) it is the first item of a backup
volume. Its name and address vary with the installation settings at backup
time. One may obtain the address within the afio archive by
afio -t /dev/cdrom 2>/dev/null | head -1
which should reply something like
u/thomas/scdbackup/scdbackup-0.8.6/tmp/ASKME_1000.1.z
(omit 2>/dev/null in insatisfying reply.)
The script should run on nearly any shell that is around but it has the
disadvantage to be relatively slow. With multilevel backups of large disks
one may have to wait minutes for an answer.
Therefore a binary program is provided which can read the script and perform
the same searches much faster. Also it can search for text pieces (like fgrep)
or for regular expressions (like grep). This program usually is called by
executing command scdbackup_askme .
See text scdbackup_askme_help for usage information.
Access Permissions
Initially only the installing user and the superuser have permission to run backups by help of a scdbackup installation. In hostile or even just weakly controlled environments one should unconditionally stay with these settings and eventually install independend copies of scdbackup for additional users.
In the case of a strictly personal computer it may seem tolerable to have the superuser using the scdbackup installation of the desktop user. But one should be aware, that entertainment components of web browsers and other programs impose a risk that the desktop user might get overtaken by intruders.
Nevertheless it may be desirable to have a single scdbackup installation which can be shared by several users. In this case the superuser should install but absolutely not use that installation. In case of permission scheme "group_only", the superuser has to set the desired group membership of the installation (e.g. by chgrp -R ).
The script ./SET_PERMISSIONS allows the installing user to loosen the
access restrictions. The script expects as only argument one of the following
permission scheme names :
owner_only this is the default described above
group_only grants group members the same rights as the installing user
anybody allows any user to run backups but tries to keep them
from spoiling the configuration. Not really safe.
custom prepared name for a custom permission scheme:
edit SET_PERMISSIONS and search "Change this according to"
unprotected as the name says : totally unprotected and totally unsafe
Those settings are permanent and get restored whenever ./CONFIGURE_CD or ./CONFIGURE_DVD are run.
Restore the most secure settings by entering the scdbackup installation directory and executing :
./SET_PERMISSIONS owner_only
Inner details - Appendice
See table of content at the beginning ot this text.
Appendix DIRECTORIES :
Since version 0.8.5 the components of the scdbackup system are stored in several sub directories of the installation directory. Especially those files which need to be written during daily operation of scdbackup or during configuration are now separated from those files which usually need not to be changed.
After unpacking the installation directory scdbackup-0.8.6 contains these data files :
TIMESTAMP the timestamp of the tar archive cd_get_variables reads the configuration for CD
dvd_get_variables reads the configuration for DVD scdbackup_switcher dispatcher script for user commands and language choice
A few symbolic links refer to files in the the ./doc directory : COPYING LIESMICH README
These directories are created below scdbackup-0.8.6 :
aux a few parts which do not fit into any other category bin binary programs, resulting from C sources cmd user accessible commands conf configuration files and custom intermediate scripts doc documentation inst configuration scripts logs list files with checksum records scripts worker scripts of the scdbackup system (for bash or ksh) src C sources which lead to the programs in ./bin tmp files created during daily usage
For using scdbackup at least the directories ./logs and ./tmp have to provide write permissions. There are configuration files which allow to redirect them to other addresses outside of scdbackup's installation directory. Therefore it is generally advised not to address subdirectories of scdbackup directly but to determine their actual address by $(scdbackup -where xyz) . See also the examples throughout this text.
Older applications which directly use inner parts of scdbackup will not find
those programs and scripts any more. Applications which restricted themselves
to commands scdbackup* and sdvdbackup* and did not access configuration
files directly are not affected.
To ease the transition of affected applications there is a script
./COMPATIBILITY_LINKS in directory ./inst which creates a lot of symbolic
links within directory scdbackup-0.8.6 . This makes available all programs
which in older documentation have been advertised for use.
Eventually missing links may be created by hand like this :
cd $(scdbackup -where)
ln -s $(scdbackup -where scripts)/cd_afio_command cd_afio_command
Nevertheless i advise you to migrate eventual applications to the new directory structure.
For mere safety reasons, a normal user's scdbackup installation should not be FHS compliant. FHS mainly addresses system-global software.
To get a scdbackup installation of the superuser conformant to FHS, you would have to scatter it over your system so it matches the following settings
Configuration file : Content : $HOME/.scdbackup_dir /opt/scdbackup-0.8.6 /opt/scdbackup-0.8.6/scdbackup_conf_dir /etc/opt/scdbackup-0.8.6
/etc/opt/scdbackup-0.8.6/scdbackup_tmp_dir_value /var/opt/scdbackup-0.8.6/tmp /etc/opt/scdbackup-0.8.6/scdbackup_logs_dir_value /var/opt/scdbackup-0.8.6/logs See below appendix CONFIGURATION for the meaning of these files.
In order to obey the FHS imperative "Programs to be invoked by users must be located in the directory /opt/<package>/bin" one may also move the content of the ./cmd directory to ./bin .
Appendix CONFIGURATION :
These three configuration files are used by both CD and DVD subsystem :
$HOME/.scdbackup_dir the address of the scdbackup-directory
(such a file is created by ./ADD_USER)
May be overridden by shell variable SCDBACKUP_DIR
$HOME/.scdbackup_lang the language to be used in dialog. Currently only
en (english) and de (german) are supported.
Any other content will default to en .
May be overridden by shell variable SCDBACKUP_LANG
$HOME/.scdbackup_userid only needed if shell variable EUID is not set
automatically by the shell. This file may contain
a unique decimal number to distinguish this user
from any others of the system. The configuration
scripts will prompt for a number and create such
a file if they detect $EUID empty.
Override this file or EUID by SCDBACKUP_USERID .
(Note that this does not give you any other user's
permissions or privileges.)
The following configuration files get created by ./CONFIGURE_CD in directory
./conf
(DVD configuration files are listed further below)
This address may be overridden by the content of file
$(scdbackup -where)/scdbackup_conf_dir
resp. variable SCDBACKUP_CONF_DIR. Its actual address may be queried by
scdbackup -where conf
Each of the files *_value may be overridden by a shell variable. This
variable has the same name as the filename's beginning but is written in
capital letters. But note that an empty variable does not override anything.
Example:
Override files scdbackup_scsi_adr_value , scdbackup_speed_value and
scdbackup_buffer_value to use the cruelly fast IDE writer which chokes
my system if i use a file buffer at speed 12 :
export SCDBACKUP_SCSI_ADR="1,0,0"
export SCDBACKUP_SPEED="12"
export SCDBACKUP_BUFFER="-"
scdbackup_scsi_adr_value the address of the recorder
(e.g. 3,0 , ATA:1,0,0 , /dev/hdc)
scdbackup_speed_value the recording speed (e.g. 12)
scdbackup_blanken_value wether to blank the CD before writing:
"0"=no , "1"=yes (mode will be "fast")
or the blank= option for cdrecord:
"all" , "fast" (see ./cdrecord_prog blank=help)
or to perform blanking during the write run rather
than in a separate blank run :
"inline_all" , "inline_fast"
The text may contain all three arguments for
scdbackup_blank. Like: all - CD-RW
scdbackup_use_prog_value for CONFIGURE_CD to remember the answer to the
question which creates the next four files:
scdbackup_mkisofs_value name and options of the ISO9660 file system
generator (e.g. mkisofs -J -l -L -R)
scdbackup_path_list_value eventually contains cd_backup_planer options for
the use of the -path-list feature of modern mkisofs
versions. Among others there is the address of the
temporary list file which will buffer the path-list
for mkisofs.
Note that the text "mkisofs_path_list" gets expanded
to "mkisofs_path_list"_$EUID and that "=EUID=" gets
converted to $EUID. (This does only apply to reading
the file but not to evaluating SCDBACKUP_PATH_LIST)
scdbackup_iso_filetypes_value eventually contains options for cd_backup_planer
which may allow other file types than "data" and
"dir" to be handed over to mkisofs. Older mkisofs
aborts on symbolic links, though.
scdbackup_cdrecord_value name and fixed options of the CD recording program
(e.g. cdrecord -v )
scdbackup_cdrecord_opt_value additional options of the CD recording program
to be used with runs that write data but not with
the separate blanking run. (e.g. -fs=8m -eject)
scdbackup_split_dir_value for CONFIGURE_CD to remember the answer to the
question which creates the following two files :
scdbackup_large_file_iso_value options for cd_backup_planer concerning
handling of large files when planning an ISO
format backup.
Note that the text "=EUID=" gets converted to
the userid number. (This does only apply to reading
the file but not to $SCDBACKUP_LARGE_FILE_ISO)
scdbackup_large_file_afio_value
As scdbackup_large_file_iso_value with afio format.
Does not apply to scdbackup_sys.
scdbackup_afio_value name of a program call-compatible to afio plus
eventual fixed options for writing. The program must
recognize option "-Z" for compression and "-" as
target name for stdout. It has to expect the
addresses of input files at stdin. (see: man afio)
Default value is : afio -o
scdbackup_buffer_input_value for CONFIGURE_CD to remember the answer to the
question which creates the following two files:
scdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs)
scdbackup_afio_compress_value wether to enable compression with afio :
0 = no, 1 = yes, - = compress if buffer file is used
(default is - )
scdbackup_z_cap_value estimated capacity for compressed afio. If empty
or 0 then cd_dir_afiosize is used.
scdbackup_sys_excl_value contains a list of directories to be excluded from
the system backup.
scdbackup_t_lock_value wether to use the locking protocol:
1 = yes, store lockfiles in scbackup directory
/path = yes, store lockfiles in path (e.g. /tmp )
anything else means no locking
scdbackup_bin_sbin_root_value parent directory for installation of commands
scdbackup_is_configured indicates by content "yes" that scdbackup was
configured sucessfully for CD.
Further configuration files are not adjustable by ./CONFIGURE_CD but may be changed or overridden to customize scdbackup's behavior :
scdbackup_use_shell_value contains the path of a bash compatible interpreter.
scdbackup_media_cap_value maximum capacity of media used with afio archives.
If an afio archive exceeds this size then it is
cut on the first CD and continued at a new one.
(set initially to "650m" by CONFIGURE_CD)
This size is also used for blank option "shred".
scdbackup_max_size_value default volume size if option -max_size is omitted.
(640m if file is missing and variable not set)
scdbackup_blank_speed_value the speed for blanking CD-RWs (e.g. 4) in case
it differs from recording speed. (This file is not
created by ./CONFIGURE_CD and usually not needed.
Before my CRW8424S deceased completely it was able
to blank with speed 4 but to write only with 2.)
scdbackup_make_checksum_value contains options for generating MD5 checksums.
This file is initially created by CONFIGURE_CD but
its content may be set to a single "-" to prevent
checksum computation.
scdbackup_checksum_list_value address of the file containing checksum records.
(if missing : .../scdbackup-0.8.6/logs/backup_log )
scdbackup_verify_adr_value default for data_source with scdbackup_verify
(if missing : /dev/cdrom )
scdbackup_keep_buffer_value wether to keep the eventual buffer file with ISO
image or afio archive to survive until the next
volume gets written. Content "1" means yes.
scdbackup_options_with_iso_value additional options for cd_backup_planer
which shall take effect with any backup that
produces ISO9660 file systems.
scdbackup_options_with_afio_value additional options for cd_backup_planer
which shall take effect with any backup that
produces afio archives.
The following configuration files get created by ./CONFIGURE_DVD in directory
./conf
(CD configuration files are listed above)
This address may be overridden by the content of file
$(scdbackup -where)/scdbackup_conf_adr
Like with CD configuration each of the files *_value may be overridden by a
shell variable.
E.g.: Override files sdvdbackup_dev_adr_value and sdvdbackup_speed_value :
export SDVDBACKUP_SCSI_ADR="/dev/sr2"
export SDVDBACKUP_SPEED="1"
sdvdbackup_dev_adr_value the device address of the recorder (e.g. /dev/dvd)
Eventually used with growisofs.
sdvdbackup_scsi_adr_value the SCSI address of the recorder (e.g. 3,0)
Eventually used with cdrecord-ProDVD.
sdvdbackup_speed_value the recording speed (e.g. any, 1)
sdvdbackup_blanken_value wether to blank the DVD before writing:
"0"=no , "1"=yes (mode will be "fast")
or the blank= option for sdvdbackup_blank :
"all" , "fast"
The text may contain all three arguments for
scdbackup_blank. Like: all - DVD-RW
sdvdbackup_use_prog_value for CONFIGURE_DVD to remember the answer to the
question which creates the next two files:
sdvdbackup_mkisofs_value name and options of the ISO9660 file system
generator (e.g. mkisofs -J -l -L -R)
sdvdbackup_path_list_value eventually contains cd_backup_planer options for
the use of the -path-list feature of modern mkisofs
versions. Among others there is the address of the
temporary list file which will buffer the path-list
for mkisofs.
Note that the text "mkisofs_path_list" gets expanded
to "mkisofs_path_list"_$EUID and that "=EUID=" gets
converted to $EUID. (This does only apply to reading
the file but not to $SDVDBACKUP_PATH_LIST)
sdvdbackup_iso_filetypes_value eventually contains options for cd_backup_planer
which may allow other file types than "data" and
"dir" to be handed over to mkisofs. Older mkisofs
aborts on symbolic links, though.
Default : -result_filetypes data:dir:arg_redirect
sdvdbackup_cdrecord_value name and fixed options of the DVD recording program
(e.g.
/u/test/scdbackup-0.8.6/scripts/growisofs_wrapper )
sdvdbackup_cdrecord_opt_value additional options of the DVD recording program
to be used with runs that write data.
(e.g. -final_eject bufsize=64m)
sdvdbackup_split_dir_value for CONFIGURE_DVD to remember the answer to the
question which creates the following two files:
sdvdbackup_large_file_iso_value options for cd_backup_planer concerning
handling of large files when planning an ISO
format backup.
Note that the text "=EUID=" gets converted to
the userid number. (This does only apply to reading
the file but not to $SDVDBACKUP_LARGE_FILE_ISO)
sdvdbackup_large_file_afio_value
As sdvdbackup_large_file_iso_value with afio format.
Does not apply to sdvdbackup_sys.
sdvbackup_afio_value name of a program call-compatible to afio and
eventual fixed options for writing.
Default value is : afio -o
(See above scdbackup_afio_value for more details)
sdvdbackup_buffer_input_value for CONFIGURE_DVD to remember the answer to the
question which creates the following two files :
sdvdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs_dvd)
sdvdbackup_afio_compress_value wether to enable compression with afio :
0 = no, 1 = yes, - = compress if buffer file is used
(default is - )
sdvdbackup_z_cap_value estimated capacity for compressed afio. If empty
or 0 then cd_dir_afiosize is used.
sdvdbackup_sys_excl_value contains a list of directories to be excluded from
the system backup.
sdvdbackup_t_lock_value wether to use the locking protocol :
1 = yes, store lockfiles in scbackup directory
/path = yes, store lockfiles in path (e.g. /tmp )
anything else means no locking
sdvdbackup_bin_sbin_root_value parent directory for installation of commands
sdvdbackup_is_configured indicates by content "yes" that scdbackup was
configured sucessfully for DVD.
Further configuration files are not adjustable by ./CONFIGURE_CD but may be changed or overridden to customize scdbackup's behavior :
sdvdbackup_use_shell_value contains the path of a bash compatible interpreter.
sdvdbackup_media_cap_value maximum capacity of media used with afio archives.
If an afio archive exceeds this size then it is
cut on the first DVD and continued at a new one.
(set initially to "4480m" by CONFIGURE_DVD)
This size is also used for erasing (blanking) DVD.
sdvdbackup_max_size_value default volume size if option -max_size is omitted.
(4250m if file is missing and variable not set)
sdvdbackup_make_checksum_value contains options for generating MD5 checksums.
This file is initially created by CONFIGURE_DVD but
its content may be set to a single "-" or made
empty to prevent checksum computation.
sdvdbackup_checksum_list_value address of the file containing checksum records
(default: .../scdbackup-0.8.6/logs/backup_dvd_log )
sdvdbackup_verify_adr_value default for device address with sdvdbackup_verify
(if missing : sdvdbackup_dev_adr_value )
sdvdbackup_keep_buffer_value wether to keep the eventual buffer file with ISO
image or afio archive to survive until the next
volume gets written. Content "1" means yes.
sdvdbackup_options_with_iso_value additional options for cd_backup_planer
which shall take effect with any backup that
produces ISO9660 file systems.
sdvdbackup_options_with_afio_value additional options for cd_backup_planer
which shall take effect with any backup that
produces afio archives.
This file may be created by hand before running ./CONFIGURE_DVD :
sdvdbackup_use_prodvd_value wether to configure for cdrecord-ProDVD (file
content "1") or for growisofs (any other content).
This file preserves the name of the permission scheme for ./SET_PERMISSIONS . It applies to both, CD and DVD permissions :
scdbackup_set_permissions_value one of the names mentioned in paragraph
"Access Permissions" above.
The files to redirect the directories which need w-permission for daily usage apply to both CD and DVD :
scdbackup_tmp_dir_value may contain the address of a directory which will
be used instead of the ./tmp directory within
scdbackup's installation directory.
scdbackup_logs_dir_value may contain the address of a directory which will
be used instead of the ./logs directory.
Redirection of ./conf, ./tmp, or ./logs is only recommended for situations where scdbackup is installed on a read-only filesystem and the writeable directories need to be outsourced to a rw-filesystem.
One may view the effective configuration of a scdbackup environment by
scdbackup -show_configuration | less
sdvdbackup -show_configuration | less
The burn scripts scripts/cd_backup_command , scripts/cd_afio_command ,
scripts/cd_pipe_command allow the use of three intermediate scripts or
programs. If you provide executable code under one of the following file names
within the ./conf directory, it will run at the given occasion.
Each of the runs gets at least eight arguments :
$1 Address of buffer file ("-" or "" if on-the-fly)
$2 SCSI address used by cdrecord or device address used by growisofs
$3 Caller ("iso9660" for scdbackup, "afio" for scdbackup_afio)
$4 Volume number
$5 Total number of volumes
$6 scdbackup installation directory
$7 cdrecord program used (with options like -v)
(contains .../growisofs_wrapper if running for DVD) $8 formatter program used (with several options)
These program runs are prepared :
conf/volume_prompt_script
before the user gets opportunity to press Enter.
conf/volume_preburn_script
after the eventual recorder lock is refreshed and
before the eventual blank run.
conf/volume_afterburn_script
after the run of the writer program ended and
before the eventual buffer file gets removed.
This script gets additional arguments:
$9 Address of the eventual checksum list
${10} Recordname of the record written to this list
${11} Timestamp of record
${12} Size
${13} Checksum (superstition brings bad luck. U.Eco)
Make sure your shell can evaluate ${10} or use shift
to access double digit parameters (after copying all
single digit parameters needed).
conf/volume_abortedburn_script
runs instead of volume_afterburn_script if the
writer program indicates failure and the whole
backup run is to be aborted.
There may also be scripts specialized for a particular media type. If both - the general script and the specialized script - are present, then the generalized script gets executed first and the specialized script gets excecuted after that. Both scripts get the same arguments.
Specialized scripts for CD are :
conf/volume_prompt_script_cd
conf/volume_preburn_script_cd
conf/volume_afterburn_script_cd
conf/volume_abortedburn_script_cd
For DVD :
conf/volume_prompt_script_dvd
conf/volume_preburn_script_dvd
conf/volume_afterburn_script_dvd
conf/volume_abortedburn_script_dvd
Example :
Put your backup into a set of hard disk files rather than onto a set of media.
For that prepare a writer which directs the volume stream into a disk file :
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_data_to /home/archive_dir/next_volume"
Create an afterburn script which renames the disk file to a filename like "A50922.132751_volume_1_7" :
echo '#!/bin/sh' > $(scdbackup -where conf)/volume_afterburn_script echo 'mv /home/archive_dir/next_volume /home/archive_dir/"${11}"_volume_"$4"_"$5"' >> $(scdbackup -where conf)/volume_afterburn_script chmod u+x $(scdbackup -where conf)/volume_afterburn_script
Now run your backup and have "yes" answer any media prompt
yes | scdbackup_afio -max_size 1g /opt /home
Do not forget to remove the afterburn script
rm $(scdbackup -where conf)/volume_afterburn_script unset SCDBACKUP_CDRECORD
Appendix WORKERS :
The commands use a program called bin/cd_backup_planer as central component. cd_backup_planer splits a list of directories into pieces which then each fit on CD-R or other media. It generates a script with backup commands. The lists of files or directories are determined according to maximum capacities in terms of total size and number of items. Oversized directories are split automatically. Oversized files are skipped unless a "Large file split directory" was defined during configuration.
Also the program is able to generate a shell script that can print several informations about the backup. This script then is included in each piece of the backup.
Read text
doc/cd_backup_planer_help
or execute
$(scdbackup -where bin)/cd_backup_planer -help | less
to get a description of the program and its options.
To avoid problems while running mkisofs the program performs a collision test of target names and a readability test of source names. Collisions are circumvented by automatical renaming. Problematic filenames are stated explicitely. Unreadable files and directories are excluded automatically.
The command scripts (e.g. /usr/bin/scdbackup) merely determine the location of the scdbackup system and then start a master script there :
scdbackup_switcher finds out what language to use in dialog. It reports
the command name, the scdbackup version and the language
code on stderr. Then it starts one of the worker scripts.
If the only argument of the command is one of the
following, scdbackup_switcher prints a reply to stdout
and ends rather than starting the program :
-version "scdbackup" versioncode timestamp
-which the address of the effective worker script
-where the address of the effective scdbackup
installation directory. An optional second
argument may request the actual address of
a scdbackup subdirectory in case it is
redirected from its usual place. Like :
bin conf doc logs scripts tmp
(see also its usage throughout this text)
-show_configuration a list of variable names and
contents. See appendix CONFIGURATION.
-show_configuration_exported a list of export commands
for the shell. Content is quoted.
The scripts use two common shell code snippets to read the configuration settings :
cd_get_variables reads the variables which where set with CONFIGURE_CD .
(to be executed inline via dot)
dvd_get_variables reads the variables which where set with CONFIGURE_DVD
(and uses them to override CONFIGURE_CD's settings
in cd_get_variables).
The following files are stored in directory ./scripts .
scdbackup_switcher gets ordered to start one of the following scripts :
backup_iso_en plans and performs backups in ISO 9660 format.
It is used by scdbackup, scdbackup_home,
sdvdbackup, sdvdbackup_home
backup_afio_en plans and performs backups in archive format.
It is used by scdbackup_afio, scdbackup_sys,
sdvdbackup_afio, sdvdbackup_sys
blank_en blanks a single media and/or overwrites it with
random bytes. (scdbackup_blank, sdvdbackup_blank)
verify_en verifies a single media against recorded checksums.
(scdbackup_verify, sdvdbackup_verify)
askme_en starts the binary information program.
(scdbackup_askme, sdvdbackup_askme)
clear_en releases all possibly open locks of a scdbackup
installation.
show_confguration performs option -show_configuration (see above).
Common code snippets which make the command scripts less redundant :
handle_existing_script performs options like -resume and other activities
which deal with the backup script that resulted from
the previous run.
handle_planer_failure is called if the run of cd_backup_planer fails.
install_backup_script installs the new resulting backup script and makes
it ready for execution.
Size determination is crucial for planning the distribution of backup data. Besides the -internal size measure there may be used du -sk or scripts which simulate the intended backup format :
cd_dir_isosize Deprecated, is able consume enormous virtual memory :
can be used instead of du -sk to estimate more
accurately the size which a directory will have in
ISO9660 filesystem.
cd_dir_afiosize can be used to determine the size of a (possibly
compressed) afio archive made from a particular
directory. This is very slow since the archive is
actually generated, counted and discarded.
Several scripts scripts which refer to writing of media and pseudo-media :
cd_backup_command is able to write a single piece to a CD or DVD writer
as ISO9660 file system.
cd_afio_command is able to write a single piece as afio archive
to a CD or DVD writer. If a buffer file is used then the
archive contains compressed files (afio option Z).
For CDs written on the fly this isn't suitable
since the output rate of a compressor can be very
low (i.e. the compression rate can be very high).
cd_pipe_command directs stdin to a single CD or DVD without any further
formatting. I.e. data already have to be formatted.
This is used with scdbackup -pipe_to_media .
If the first argument is -multi_volume then multi volume
rules as with scdbackup_afio apply. Use this mode for
piping large archives or dumps onto several media.
cd_blank_command blanks a CD-RW using the configuration of scdbackup
and eventually starts shredder_command.
sdvd_blank_command warns of shredder_command and starts it for DVD.
shredder_command overwrites a media with random data.
afio_sencrypt a script which produces an encrypted and eventually gzip
compressed afio archive. Encryption is done by sencrypt.
See http://scdbackup.webframe.org/README_sencrypt
See also appendix CRYPTO below.
One of three modes is selectable by the first argument.
Depending on the mode a second argument may be needed :
-each_file_by keyphrase_file
afio-traditional encoding of each single file
yields a rugged archive but is quite slow.
The key phrase has to be stored in a file and the
second argument has to be its address. In case of
compression the first 10 bytes of each file stay
unencrypted to avoid attacks via known cleartext.
Unpack (uncompressed) :
afio -i -P 'sencrypt' -Q '-d' \
-Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom
Unpack (compressed) :
afio -i -P 'sencrypt' -Q '-d' -Q '-g' \
-Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom
-whole_archive_by keyphrase_file
encryption of the resulting archive as a whole.
Again the key phrase has to be stored in a file.
With uncompressed archives the first 322 bytes
stay unencrypted. With compressed ones it is 162.
Unpack (uncompressed) :
sencrypt -d -s 322 -f "$keyphrase_file" \
</dev/cdrom | afio -i -
Unpack (compressed) :
sencrypt -d -g -s 162 -f "$keyphrase_file" \
</dev/cdrom | afio -i -
-whole_archive
encryption of the whole archive. The key phrase is
requested at /dev/tty. The first 322 resp. 162
bytes of the archive stay unencrypted.
Unpack (uncompressed):
sencrypt -d -s 322 </dev/cdrom | afio -i -
Unpack (compressed) :
sencrypt -d -g -s 162 </dev/cdrom | afio -i -
If the next argument is -Z then compression is enabled.
This -Z is eventually provided by scdbackup if
compression is configured (s*_afio_compress_value).
Any further arguments are forwarded to afio -o.
Variable SCDBACKUP_SENCRYPT_OPTIONS may be used to
override the preset sencrypt options. E.g. to avoid any
unencrypted head (and invite known-plaintext-attacks):
export SCDBACKUP_SENCRYPT_OPTIONS="-s 0"
star_as_afio_wrapper a script which replaces usage of program afio by usage of
program star. This will then produce star-archives rather
than afio archives. About encryption see also appendix
CRYPTO and http://scdbackup.webframe.org/README_sencrypt
Options recognized (-Z is given by scdbackup as needed):
-Z compress by help of star -z
-acl record ACL information by star artype=exustar -acl
artype=archive_type_of_star
choose a particular archive type and thus disable
exact -dump mode of star. E.g. artype=gnutar
See option "artype=" in man page of star.
This option may be overridden by -acl.
-encrypt
encrypt the whole archive by sencrypt and a
prompted key phrase.
Unpack (uncompressed):
sencrypt -d -s 752 </dev/cdrom | star -x
Unpack (compressed):
sencrypt -d -g -s 162 </dev/cdrom | star -x
-encrypt_by keyphrase_file
encrypt the whole archive by sencrypt and a key
phrase stored in a file.
Unpack (uncompressed):
sencrypt -d -s 752 -f "$keyphrase_file" \
</dev/cdrom | star -x
Unpack (compressed):
sencrypt -d -g -s 162 -f "$keyphrase_file" \
</dev/cdrom | star -x
-fifo
use star option -fifo despite it obscures errors
errctl=filename
finely adjustable error handling instructions for
star (see man page: option "errctl=")
-help explain options, predict unpack command, abort.
Any other arguments are ignored. No whitespace is allowed
in filename arguments (they have to be a single word).
Variable SCDBACKUP_SENCRYPT_OPTIONS applies (see above).
I got my copy of star from
http://cdrecord.berlios.de/old/private/star.html
compiled it and installed the binary in /usr/bin.
Online man page of star
http://cdrecord.berlios.de/old/private/man/star.html
Usage example (exustar allows files larger than 2 GB):
export SDVDBACKUP_AFIO_COMPRESS="1"
export SDVDBACKUP_LARGE_FILE_AFIO=" "
export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -acl"
sdvdbackup_afio ...
Applies to sdvdbackup_sys, too.
List archive's table of content:
gunzip </dev/cdrom | star -tv
growisofs_wrapper a script which replaces the call of cdrecord in
cd_backup_command and cd_afio_command. It calls program
growisofs.
It recognizes the arguments -final_eject and
-no_final_eject which control disk tray behavior after
growisofs did its unload-reload cycle.
For other options see output of:
$(sdvdbackup -where scripts)/growisofs_wrapper -help
pipe_data_to a script which may be used rather than cdrecord or
growisofs_wrapper to direct the backup to an arbitrary
file address. It ignores any but its first and its last
argument. The first one is the target address
("-" = stdout), the last one is the the data source and
will be provided by scdbackup. If it is not "-" (= stdin)
and not the address of an existing usable file, the
script will print a notice on stderr and end with exit
value 0. Example (write to a 250 MB QIC tape device) :
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_data_to /dev/rmt0"
scdbackup_afio -max_size 240m ...
pipe_to_raedchen a script like pipe_data_to which uses raedchen as filter
in order to show a progress counter or to limit troughput
or to ease i/o buffer load. Mandatory arguments :
$1 = the target like with pipe_data_to.
$2 = the -step value for raedchen (0=quiet).
$3 = the -baudlimit value for raedchen (0=unlimited).
$4 = the flush option for raedchen (-flush or -).
Example (write to DVD-RAM without clogging disk I/O):
export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 1m 0 -flush"
pipe_to_cdrecord_sao a script which circumvents the need to have a buffer
file with cdrecord in -sao mode (e.g. with DVD). This is
done by announcing the media_cap size to cdrecord and
padding shorter volumes up to that maximum size.
The script expects as first argument the address of the
cdrecord-compatible program which it shall use in -sao
mode. The last argument may either be "-" for stdin or
the address of a readable file. Most other options are
handed over to that program and are followed by a last
argument "-".
For an example see below Appendix DVD, cdrecord-ProDVD.
pipe_to_ssh a script like pipe_data_to which forwards the input to a
remote installation of scdbackup via SSH. The first five
arguments are not ignored but define the remote service :
hostname login with_passwd remote_command remote_option
Example (for writing DVD on host ts2 via user thomas) :
.../pipe_to_ssh ts2 thomas 1 sdvdbackup -pipe_to_media
The value "1" of with_passwd indicates that the login
will ask for a password. This triggers a 15 second wait
period and a (non-password) user prompt before ssh is
executed. See also appendix NET.
pipe_to_null a script which only reports its start and then ends
unless its last argument is "-". In that case it pipes
stdin to /dev/null and ends only when stdin is closed.
It ignores any other arguments and therefore is suitable
to replace cdrecord or growisofs_wrapper when no media
shall be burned.
noop like pipe_to_null but ends immediately in any case.
mxn_wrapper a writer script that multiplies and/or permutes backup
volumes.Its arguments form three groups separated by ".":
type inner_cp media_cp [second_buf permutation_keys] .
writer_program [and options] .
[ignored arguments] buffer_file
It only works with buffer file. The backup volume gets
copied inner_cp times on each media. A number of media_cp
media is filled with this multiple volume. Writing is
done by the writer program specified in the second group
of arguments. In the last group only the last argument
is of interest. For [second_buf permutation_keys] see
appendix REDUNDANCY below.
type is either "DVD" or "CD".
ask_user performs user interaction when a new media is required.
Does not apply to archive overrun handling.
return_wrapper a script which catches the exit value of the ISO
formatter program and writes it to a file in ./tmp
Two special pieces of shell code are executed inline by the example scripts to control the locking protocol:
cd_lock_skript locks the result script
Expects the following variables to be set:
$t_lock ("1"=locking protocol activated)
$skript (name of result script)
$bdir (name of scdbackup directory)
$slock_time (lifetime of locking)
Sets:
$s_id (id of the lock)
a trap which at end of the script releases the
lock on the result script.
cd_lock_skript_and_cdrecorder refreshes the lock on the result script and
locks the CD recorder.
Expects the following variables to be set:
all which are expected by cd_lock_skript
$s_id (id from cd_lock_skript)
Sets:
$c_id (id of the CD recorder lock)
a trap which at end of the script releases the
lock on result script and CD recorder.
The locking protocol is implemented by the program temp_lock and several scripts. The example scripts first lock their result scripts. After the result script is generated, the CD recorder gets locked. Locking is implemented by data which are stored in lockfiles.
Lockfiles have the name *.lock and guard the following objects : ${prefix}automatically_created*.lock backup scripts with user number and
"-cd" or "_dvd" indicating media
${prefix}currently_blanking.lock indicates an eraser run
${prefix}cdrecorder.lock all burner drives (together)
The text ${prefix} is either the address of the scdbackup installation's ./tmp directory or ${path}/scdbackup_ if another directory ${path} has been configured to coordinate locking.
lock filename seconds [factor] tries to obtain a lock on filename.lock .
The lock has a maximum lifetime given by seconds/factor
seconds may be a bc expression. It is important to
memorize the standard output of this command. It is
needed for releasing a lock.
Example:
id=$(lock cdrecorder 1800+7200 2)
locks for at most 1800+7200/2 = 5400 seconds
release filename id releases the lock on filename.lock . id is the text which
has been printed by lock when the lock was obtained.
Example (release the lock of the example above):
release cdrecorder $id
refresh filename id seconds [factor] prolongs the lifetime of a lock if
it is less than the given time (seconds/factor). Also
a check is made wether the lock is still valid. (If
invalid then the example scripts abort or try to
regain the lock.)
lock_state prints the active lock files
In directory ./bin there are several programs which get generated by inst/COMPILE from C source files located in ./src .
cd_backup_planer the central planning program. It also fulfills a lot
of other tasks. See doc/cd_backup_planer_help .
raedchen counts bytes and shows the count from time to time
on standard error. A spinning bar is shown like
on a booting SUN 4 workstation.
Also it divides and concatenates large archives
if necessary.
(see: ./raedchen -help )
saskme performs accelerated evaluation of ASKME files.
temp_lock manipulates temporary locks in special lock files and
gives informations about their state.
(See temp_lock -help )
Workers and binaries produce backup scripts in scdbackup's tmp directory.
These scripts are started for performing the backup resp. restarted for
resuming an interrupted backup.
The script names are composed by the particular commands as follows
scdbackup , scdbackup_home :
automatically_created_backup_script_${userid}${dvd}
scdbackup_afio :
automatically_created_afio_backup_script_${userid}${dvd}
scdbackup_sys :
automatically_created_sys_backup_script_${userid}${dvd}
${userid} is either the system userid number or the number set by
$HOME/.scdbackup_userid
${dvd} is empty for CD backups and "_dvd" for DVD backups.
Each script may be accompanied by a file ${scriptname}.pieceno which records
the current volume number of the ongoing backup. This number is used if a
backup shall be resumed and no explicit volume number is given option -resume.
The file ${scriptname}_emerging hosts the emerging script content during the
planning phase of the backup. It is later renamed to ${scriptname} if the
planning process was successfull. A very small placeholder file remains.
This procedure shall protect resumable backup scripts from being erased by
inadverted starts of of the planning process cause by a misspelled option
-resume as long as the option begins with "-".
Configuration is done by a hierachy of scripts which are located in ./inst .
Intended for execution by the user :
CONFIGURE (yet quite unnecessary) master script
CONFIGURE_CD main script for CD configuration
CONFIGURE_DVD same for DVD
SUPERUSER_CONFIGURE to be run by the superuser after configuration is done.
Installs scdbackup_en scripts as scdbackup commands and
sdvdbackup_en scripts as sdvdbackup commands.
ADD_USER to be run by other interested users
FETCH_CONFIGURATION copies previous configuration to new installation MIGRATE_CONF_DIR adapts a configuration directory of an incremental backup
to the new scdbackup directory (rarely necessary, though).
DIET Reduces the size of the scdbackup installation without
damaging its backup functionality.
Without any arguments it removes the binaries mkisofs_prog
and cdrecord_prog, some configuration scripts needed only
for initial configuration, the complete german language
version, most of the website files, and the delivery
scripts. It also compiles the binaries to minimal size
and then removes their source code.
Optional argument -remove_inst removes all configuration
scripts, optional argument -remove_doc removes all
documentation files. Optional argument -radical does both.
Non user executable :
CONFIGURE_VERSION sets variable scdbackup_conf_version CONFIGURE_FUNCTIONS some functions common to both CONFIGURE_CD and _DVD :
evaluate_local_mkisofs() , write_mkisofs_config()
ask_text() , ask_text_vf() , write_value()
default_setting() , remove_file() , normalize_input()
check_for_tmp() , write_large_file_options() ,
write_buffer_conf()
COMPILE inline code produces the binaries from C source
PROMPT_USERID asks for a user id number if $EUID is empty
Appendix RECOVERY
If you need to restore files after a minor mishap then you may use the programs which are appropriate for the backup format. Namely mount , cp -a for ISO-9660 filesystems
afio -i for afio archives star -x for star archives
For details see the appropriate man pages. For instructions about restore of incremental backups see doc/examples.html#restore_incr
The situation becomes more complicated if your whole system passed the way. You will then have to replace eventually faulty hardware and to start from scratch. There are several approaches which depend on your preparations and on the particular situation after the disaster.
With all those approaches, the backups of user data and optional software packages will be restored after the system setup is completed.
There are two main situations after a disaster :
- Your hardware can be revived resp. replaced by sufficiently compatible components. Then you may strive for a flat restore of your system.
- New hardware has to be installed which would have overextended your sysadmin skills if it had replaced the old hardware in a regular way without disaster. Then you have to install a new operating system and the system backup is usable mainly as source for configuration files or custom system add-ons. This illustrates the importance of a lean system backup. Try to put anything non system dependend into other backups. (See "Superuser Planning")
There are two preparation strategies supported by scdbackup :
- scdbackup_sys which records the non-volatile file objects of the system
- complete star backups of the system partitions
It can hardly be a mistake to make backups of both kinds. There are several disaster recovery systems available which might serve the same purpose. You should consider to use them too. The more choice you have in case of disaster the better are your chances to recover seamlessly.
For any disaster recovery you have to record your partition layout. E.g. print
the output of
set -x ; hostname ; date ; fdisk -l ; mount ; df ; cat /etc/fstab ; set +x
on paper and store the sheets with the backups.
Only a sufficiently similar layout of the system partitions and identical
addresses of the data partitions will allow you a flat recovery of the system.
For making use of a backup made with scdbackup_sys you will need installation
media of your operating system. Copies can be made like :
dd if=/dev/cdrom of=/somewhere/bufferfile
dd if=/somewhere/bufferfile | scdbackup -pipe_to_media
resp. for DVD :
dd if=/dev/dvd of=/somewhere/bufferfile
dd if=/somewhere/bufferfile | sdvdbackup -pipe_to_media
rm /somewhere/bufferfile
Usually scdbackup_sys produces one or more media with single afio archives. Each of them is to be unpacked by afio -i . If a media overflow had occured at backup time then you will have to concatenate the two media contents as shown below for star or above in paragraph "afio Archives". (It pays off to redo an overflown system backup with smaller -max_size rather than to accept coupled media pairs.)
With a set of scdbackup_sys media, you have these two options :
- Installation of the old operating system to the old or to sufficiently compatible hardware. Install the operating system in a minimal version but with all desired partitions and filesystems. Then boot a rescue system and copy the system backup over the minimal system installation. A suitable rescue system for x86-Linux computers is RIP. It contains all extraction software mentioned above. Use the rescue systems's bootloader (e.g. GRUB) to boot the restored system, and then run the install procedure of the system's actual bootloader (e.g. lilo) to restore the original boot facilities. You will also have to make the necessary configuration changes to acommodate your new hardware. If booting would not be possible with the old configuration, then you may try to change it from the rescue system or by your Linux' installation tool.
- Installation of a new operating system. Configure and tune it as near to your needs as you can. If you need configuration parameters, refer to the configuration files in the system backup. It is convenient to unpack it in some temporary subtree of the new filesystem. The system backup may also be the source of files which you cannot find in the user data backups because they are not under /home or /opt.
If the system backup was made as star archives then the procedure stays the same. Nevertheless star offers a more complete system backup if you allow it to take care of the whole system partition.
To make use of a star level=0 backup of the system partitions you have to run system backups by this command pipeline rather than by scdbackup_sys :
star -c -xdev -acl -link-dirs level=0 -C / . \ | gzip | sdvdbackup -pipe_to_media -multi_volume
Consider seriously to do this in single user mode or from a running rescue system like RIP with scdbackup additionally installed on it (see below).
Be prepared to insert additional media if one DVD is not enough to take
the compressed archive. In that case you will have to practice concatenation
without help of scdbackup. Like this command for two media :
( dd if=/dev/hdc bs=2048 count=2293760
echo "insert next DVD" >&2 ; read dummy
dd if=/dev/hdc bs=2048 count=2293760 ) | gunzip | star -t -v
The number 2293760 is for DVD (4480 MB). With CD use 332800 (650 MB).
With a single DVD life is simpler :
gunzip </dev/dvd | star -t -v
(For real restore use star option -x rather than -t )
If /boot or /usr are separate filesystems then one should make such a backup
with those partitions too :
star -c -xdev -acl -link-dirs level=0 -C /boot . \
| gzip | scdbackup -pipe_to_media -multi_volume
star -c i-xdev -acl -link-dirs level=0 -C /usr . \
| gzip | scdbackup -pipe_to_media -multi_volume
This allows :
- Flat restore of a complete star backup which demands sufficiently compatible hardware. If the disk was replaced or the partitioning was lost, then you have to partition it first and install filesystems of the appropriate type. This can either be done by a consumer Linux distribution's installation tool or by a rescue system. Then a rescue system is booted which can extract star archives (x86 use RIP). Again, you will have to adapt the restored system to eventual hardware changes. Use the rescue systems's bootloader (e.g. GRUB) to boot the restored system, and then run the install procedure of the system's actual bootloader (e.g. lilo) to restore the original boot facilities.
- Installation and configuration of a new operating system and use of the star archive as source for mainly configuration files like above with scdbackup_sys.
If you have Intel/AMD/x86 hardware, then get an ISO-9660 image of RIP from:
http://www.tux.org/pub/people/kent-robotti/looplinux/rip/
and burn it to CD like :
dd bs=2048 if=/somewhere/RIP-14.4.grub.iso.bin | scdbackup -pipe_to_media
Next make an emergency drill: insert the RIP CD, reboot, choose RIP in the
GRUB menu which will appear, login as "root" (no password requested), and
check your archives for readability by
afio -tv /dev/hdc
resp.
gunzip </dev/hdc | star -t -v
At this occasion you should also record the MBR of your hard disk.
See (when RIP is up) :
/usr/doc/RIP/backup-mbr.txt
There is a matching binary-only version of scdbackup-0.8.6 for RIP at
http://scdbackup.webframe.org/scdbackup-0.8.6-x86-rip.tgz
which you may download and put on some suitable media. It will help you with
unpacking encrypted backups, damaged high-redundancy backups, multivolume
archives. You may also make after-disaster backups of still readable data
partitions on your hard disks.
Just unpack after booting RIP
( cd / ; tar xzf .../scdbackup-0.8.6-x86-rip.tgz )
Most features described in this README should run out of the box. The only
mandatory configuration effort is to find out the addresses of the DVD
and/or CD burner. (Try: eject /dev/hdX , cdrecord dev=ATA -scanbus )
Without further configuration you may just set the configuration variables to
that address. Like
export SCDBACKUP_SCSI_ADR="/dev/hdd"
scdbackup /=/mnt
or
export SCDBACKUP_SCSI_ADR="ATA:1,0,0"
scdbackup /=/mnt
or for DVD
export SDVDBACKUP_DEV_ADR="/dev/hdd"
sdvdbackup /=/mnt
For verification you will give the device address explicitely and avoid the
awful behavior of the cdrom driver at end of disk :
scdbackup_verify /dev/hdd -auto_end
sdvdbackup_verify /dev/hdd -auto_end
You may run the configuration scripts, too
cd /opt/scdbackup/inst
./CONFIGURE_CD
./CONFIGURE_DVD
See also text http://scdbackup.webframe.org/scdbackup-0.8.6-x86-rip.txt
If you do not use x86-Linux, then you should look for a standalone-tool which
can do the following for your hardware and operating system :
- mount the desired filesystem types in read-write mode
- unpack afio resp. star archives
- for flat star restore : format disks and filesystems
Alternatively you can try to perform the unpacking in the single user mode of a preliminarily installed operating system.
Appendix AUTOMAT
Although scdbackup is a software which is intended to be used in scripts, its normal usage model is not for unattended operation. This is due to the fact that with common burner hardware it is necessary to insert a media and to confirm this preparation by an input line at the control terminal of the backup run.
The input lines are buffered, so it is no problem to give such an OK for the first volume in advance. Just press the Enter key again immediately after you started scdbackup.
This does not help, of course, if there is no control terminal, because scdbackup gets started as a background process (by cron, window managers, etc). In such a case you will have to provide media management. You are lucky if you only have to expect a single volume.
The input requests of scdbackup expect as answer a newline character at stdin. Other (printable) characters are ignored. So you may avoid the input requests by a pipe like this :
yes | scdbackup ...
A filter which answers automatically once and then waits for user input :
( echo ; while read dummy ; do echo ; done ) | scdbackup ...
If there is more than one volume then some kind of media change has to be
performed before the next volume can be written. To trigger this, you may
provide intermediate scripts as described in Appendix CONFIGURATION :
conf/volume_prompt_script
conf/volume_preburn_script
conf/volume_afterburn_script
conf/volume_abortedburn_script
You will have to decide what action to perform at which occasion. If you
want to send mail to a human operator and wait for a reply, then do this
in conf/volume_prompt_script.
You possibly want to control the whole backup run by own means. See script outer_loop for an example how to break down the run into single media steps.
A note about cron :
Cron runs are known to happen in a very sparse shell environment.
scdbackup relies on the existence of environment variables $HOME and $EUID
(if it was present during configuration).
Therefore it is a good idea to wrap scdbackup in a script which ensures those
two variables are set properly. One should also not rely on the preset variable
$PATH but replace its content by the text that is replied by
echo $PATH
in an interactive shell session of that user.
Example script for user "root" :
#!/bin/sh
test "$HOME" = /root || export HOME=/root
test "$EUID" = "" && export EUID=0
export PATH="/usr/sbin:/bin:/usr/bin:/sbin:/usr/X11R6/bin"
yes | sdvdbackup -pacifier -off ...your.arguments...
Appendix VERIFY
The commands s*backup_verify compare the byte stream from a backup volume with checksum records. These records may be obtained from a record list file or from the end of the backup volume itself. The latter is possible only with backups written by scdbackup 0.8.3 or younger.
Checksum computation, recording and comparison is usually done by program cd_backup_planer. This appendix specifies the checksum data formats on disk and backup media as well as redundancy and permutation formats.
Checksum Records
Checksum Tags
Block Checksum Lists
Multiplied Volumes
Permuted Volumes
Checksum Records
A checksum record consists of four words and a newline character :
name timestamp size md5
name may consist of up to 80 characters.
"1_1"
timestamp is of format YYMMDD.hhmmss with "A0" = 2000 , "A1" = 2001 ,
"B0" = 2010 , ... like: A40904.230615 .
size is a decimal number and may become quite large.
Expect more than 32 bit. Like
"4529208064"
md5 is a MD5 checksum compatible with programs like md5sum.
It is represented as printable lowercase hex number. Like
"4e64fdbb3