As many DP4 installers have had problems designing safe backup/restore mechanisms for DP4 databases using our current database utilities, especially in situations where 24*7 operation is necessary, we have reworked these utilities with a view to making them safer and easier to use. This document describes the changes on a program by program basis and discusses implementation details.
Once, when DP4 was first designed, computer hard disks were about 10 Megabytes in size, or 40 Megabytes for a big machine. DP4 databases were rarely more than 5 or 10 Megabytes in size. At the end of the day the operator would back the database to 5 or 6 diskettes (5.25 inch floppies back then). The whole process took about ten minutes. The DBBACKUP program prompted users to load and unload diskettes,and told them how to label them. It also prevented disks from earlier backups from being reused for a while.
Now DP4 databases are often in excess of 3 or 4 Gigabytes in size now, and are rarely if ever backed up to diskette. The most common scheme is to take the DP4 system down and to backup to tape. The DYNABACK program allows an open database to be backed up, however the backup format is inconvenient as the restore process is two stage - restore the backup file from tape, then restore the backup with DBRESTOR. This means that it takes longer than ideal to recover a backup. In addition there is no easy way to verify the integrity of the backup, or indeed of the database, opening up the possibility that a complete backup cycle can be gone through with an undetected corruption. We have always recommended users to run DBCHECK at least once during the backup cycle, but for large databases this can be very inconvenient.
The changes to the backup utilities are designed to fulfil various objectives:
The core backup utility remains DYNABACK, and when run interactively it continues to work as before. The new version of DYNABACK is designed to be run as a batch program, which is enabled by specifying -COPY on the command line. When you start DYNABACK like this then instead of backing the database up in BKP format it simply copies the database to a temporary database name ($$$0AC or something similar). This copy is written in the normal database directories. The copy is done using special internal functions from the database manager. This allows the database to be backed up while it is still open. The mechanism for doing this depends on DP4 rollback recovery. Please note that while DYNABACK is backing up the database the rollback file cannot be restarted, and it can therefore grow to quite a large size. It is advisable to run DYNABACK at periods of low activity.
When the backup has completed DYNABACK immediately runs DBRECOV against the backed up database to apply the backed up rollback and transaction logs. The copy of the database is then identical to the original database immediately prior to the point the backup completed.
Next DYNABACK runs DBCHECK against this database. If no corruptions are found the backup is good. At this point the backup files are renamed to DBNAME.DB1 and DBNAME.IB1 or to DBNAME.DB2 and DBNAME.IB2., and a marker is written to the original database to indicate that a successful backup has completed, and to indicate whether the B1 or B2 name has been used. The names are used alternately, because it is not safe to delete the previous backup until the current backup has completed,verified AND the database has been notified of this. Once this process is completed the previous backup is deleted if it is still present. If space permits it is advantageous to keep the current backup online, as it will permit a faster recovery if the database is lost (for example if DBRECOV fails following a system crash). The backed up database files can now be copied to tape or CD ROM. As they are in the regular DP4 database format the restore process is a single step operation - there is no need to run DBRESTOR afterwards, except to apply a log file. Alternatively a CD ROM copy of the database can be accessed in the normal way by DP4 programs, which may be useful for archive purposes.
If the DBCHECK process discovers a problem on the backed up database the bad backup is renamed instead to .IBB and .DBB. (It is not deleted automatically, as you may wish to run DBCHECK or other utilities interactively to discover more about the problem). However, as the backup is corrupt, it is very likely that there is a problem with the original database. Therefore a special flag is set in the original database. When applications are run against the database they will display a warning message indicating that the database has a probable corruption, and that DBCHECK should be run. This message will continue to be displayed by each new application until such time as DBCHECK is run, and passes, the database is successfully reorganised with REORGDB, or a backup copy of the database is restored. Normal operation is not disabled, as sometimes it is preferable (from a user point of view) to risk continue operating with a database that is corrupt, rather than disrupting the system by immediately restoring a backup. This flag also gets set if the database manager encounters a problem which causes a system error 87 (database corruption).
If your database is very large it may take too long or have an unacceptable impact on performance to run DBCHECK against the backup, though it is of course the backup and not the live database that is being checked. It is therefore possible to disable the automatic DBCHECK that takes place after a -COPY backup. You are recommended not to do this unless absolutely necessary. We have recently been able to run DBCHECK against a 3 Gigabyte database in only a few hours, so do not assume that you cannot run DBCHECK. If you do use the option to disable DBCHECK then you should transfer the backed up database to another computer where you can run DBCHECK without impacting performance of the system. If you do not run DBCHECK at least once during the backup cycle you risk losing your database. It is best to run it after every backup, otherwise you may waste time restoring a backup that itself turns out to have a corruption.
If despite the above you want to disable automatic DBCHECK as part of the backup process then specify -NOVERIFY on the command line to DYNABACK.
The backup process does not transfer database data across the network, so it may be run on a client. Note however that the DBCHECK phase does transfer data across the network, so it is best to run from the server unless the -NOVERIFY option is used.
The backup process only prevents the rollback file from being flushed while the database is actually backing up. Therefore running DBCHECK as part of the backup will not adversely affect the size of the rollback file.
The DBRECOV program continues to be used to apply a roll-back and roll-forward log after a database crash. The primary changes are as follows:
The changes to DBRECOV should mean that DBRESTOR is not normally needed. DBRECOV will let you recover the database even if there is no rollback as long as the last backup is still available. DBRECOV will run and restore the last backup even if the database could be opened, but the flag indicating genuine database corruption is set. You may be forced to run DBBACKUP again immediately after this process. This will be forced if a backup has been attempted on the corrupt database, and there is now both a .LOG and a .LBK file for the database. The backup is necessary to allow DBRECOV to restore the database again if another corruption arises. If necessary you can SETSESSN to skip this backup. In this case you must backup any LBK file before running the setsessn, and the current log file. Otherwise you risk losing changes since the last succesful backup in the event of an unrecoverable corruption.
The new DYNABACK depends on updated DBCHECK,SRVn/DP4SRVR.W32, and (for Windows) on an updated TRMW32.DLL, all included with the download.
When DBCHECK is run it sets a flag in the database header to indicate whether the database has problems or not.
The trm_run_program() function supports a new flag value: 8 causes the program to be run synchronously - i.e. the user cannot reactivate the calling program until the called program terminates.
Please take note of the paragraph below - the changes made to these programs are extensive, and we cannot guarantee that there will not be unforeseen problems with these new utilities.
Beta software is supplied without warranty and should not be used in live sites without the agreement of Itim Technology Solutions. Compatibility with an eventual final release cannot be guaranteed.The download for the new utilities will be made available as soon as testing is completed.
Beta 4.618 DP4 utilities for Windows 9x and NT/2000These new utilities are also available in the 4.5xx series for Unix/Linux or Windows. Let us know if you are interested.