FairCom — Database Backup Guide

The "dynamic dump" feature provides an administrator a safe, secure method of backing up data while FairCom DB is operational. The Administrator can schedule a dump of specific files, which may be all files necessary for recovery or a subset of them. The dump executes while FairCom DB is actively processing transactions and is transparent to users.

Dynamic Dump

FairCom DB performs a dump at first opportunity on or after a scheduled time. When beginning a scheduled dump, FairCom DB temporarily halts new transactions and starts the actual dump as soon as all active transactions complete or abort after a predetermined delay period.

The default behavior is to wait for all pending transactions to complete. This may cause a timeout status error (156 NTIM_ERR) to be logged. That error does not imply anything about the dump, other than it waited for the transaction to complete. To abort all pending transactions and start the dynamic dump immediately, the !DELAY option can be used to abort pending transactions after a specified number of seconds.

When a dynamic dump needs to abort a transaction, it puts the transaction into an error state; however it does not actually abort the transaction. With the transaction in an error state, it cannot progress or commit, so the connection that owns the transaction will abort it. Records locked in the transaction will remain locked until the connection aborts the transaction or detects that the transaction has been canceled. If this should occur, it is plausible the application will receive a TABN_ERR (error 78), indicating the dynamic dump wait has been exhausted, and the transaction was aborted.

Once the dump commences, transactions can process as usual.

Note: The dynamic dump and recovery processes are intended primarily for files under transaction processing control. Non-transaction controlled files can be dumped with certain restrictions. See Dump Files Without Transaction Control for more information.

The following sections describe the dump and recovery utilities:

Process	Utility	Explanation
Dynamic Dump	ctdump	Dumps data during server operation.
Dump Recovery	ctrdmp	Restore files to state as of last dump.
Rollback	ctrdmp	Roll the database state to an earlier time following a dump recovery.
Roll Forward	ctfdmp	Roll the database state to a later time following a dump recovery.

Scheduling a Database Backup

There are two ways to schedule dynamic dumps:

Server Configuration

The FairCom Server configuration file may be used to schedule dynamic dumps. In the file, the keyword DUMP is followed by the name of the script file defining the dump. The path to this script is relative to the server's working directory.

Dynamic Dump Utility

The dynamic dump utility, ctdump, is a separate utility for the Administrator to use at any time while the server is active.

To schedule an ad hoc dynamic dump with ctdump use the following procedure:

While FairCom DB is running, start the utility program ctdump as any other program in the environment.
Enter the password for the ADMIN administrator account.
Enter the current FairCom Server name (if assigned a different name than the default). See Basic Configuration Options for information on SERVER_NAME.
Enter the name (with path if necessary) of the dynamic dump script file.

The FairCom Server confirms that it has scheduled the requested dynamic dump.

Once a dynamic dump has completed, files may be used for Dump Recovery and/or Rollback.

ctdump - Schedule Backup Utility

Note: For a complete discussion of dynamic dumps, see Backups and Data Integrity in the FairCom Server Administrator's Guide.

Operational Model:

Client

ctdump schedules a dynamic dump on the fly. The backup definition script may be located either on the server, or passed in from the client.

Command Syntax

ctdump [-s svn] [-u uid] [-p upw] [-t script] [-b bufsiz] [-n] [-c] [-o backup_filename] [-x]

Options

-s svn - c-tree Server name
-u uid - User name
-p upw - User password
-t script - Dump script name
-b bufsiz - Use buffer size of bufsiz bytes
-c - Send dump script from client
-m - Minimize progress notifications
-n - Send progress notifications to client
-o backup_filename - Write dump stream from server to file on client
-x - Write dump stream from server to stdout

For options available when scripting a dynamic dump, see Dynamic Dump Options.

The following demonstrates example usage of this utility:

ctdump ADMIN ADMIN thescript FAIRCOMS

Secure Authentication File

This utility supports the use of an encoded password file. Encoded password files keep user IDs and passwords from plain view when the utility is used within a script file. They are created with the ctcmdset utility. The plain text form of the file should be:

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

Dump and Restore Version Compatibility

The ctrdmp utility is used to restore a Dynamic Dump and the ctfdmp utility can be used to roll forward. Occasionally an update to the FairCom Database Engine may cause an incompatibility between versions. For this reason you must use the ctrdmp from the same release from which the dump was created. It is important to save a copy of the ctrdmp utility that is compatible with each dump file. ctfdmp, ctldmp, and ctrdmp utilities display the FairCom DB version used to compile them when they are run.

Compression and Encryption

The ctdump.exe utility can accept compression and encryption utility inputs.

Windows backup compressed with 7z compression utility:

ctdump -s FAIRCOMS -u admin -p ADMIN -t dump.txt -c -x | 7z a -si backup.7z

Unix backup compressed and AES encrypted with standard utilities:

./ctdump -s FAIRCOMS -u admin -p ADMIN -t dump.txt -c -x | gzip|openssl enc -aes-256-cbc -salt >backup.gz.aes

Tip! Unix/Linux Alternative

This method also works with server-side dynamic dump backups prior to this new support and can be implemented on Unix systems using named pipes (mkfifo), and sending the dump to the named pipe.

Create a named pipe:

mkfifo testpipe

The pipe blocks until both sides are in use. So we start our compression command:

cat testpipe | gzip > backup.gz &

Now we create our dynamic dump, with "testpipe" as the dump file. (This example uses the client output, however it should work the same from the server):

ctdump -s FAIRCOMQ -u admin -p ADMIN -t dump.txt -c -o testpipe

c-tree(tm) Version 11.8.0.61131(Build-180802) Dynamic Backup Utility

Reading dump stream from server with buffer size of 100000

Dynamic Dump has been successfully written to the file testpipe.

The backup.gz file contains the dump file.

The same approach can be used to compress and encrypt (and potentially other operations provided by system utilities):

cat testpipe|gzip|openssl enc -aes-256-cbc -salt >backup.gz.aes

To extract the dump:

cat backup.gz.aes | openssl enc -aes-256-cbc -d |gzip -d >backup.ctree

Use ctrdmp to extract the files from backup.ctree.

Error Codes

The following error codes are related to dynamic dump operations:

Error Name	Error Code	Explanation
FUNK_ERR	13	Cannot determine file type. Possibly a c-tree V4.3 file?
READ_ERR	36	Failed to read file, either a corrupted or non-tree file.
TCOL_ERR	537	Transaction log collision. Two sets of transaction logs in the same directory?
FCPY_ERR	796	Immediate dump restore file copy failed.
DRST_ERR	797	Immediate dump restore failed.

Backup Script Options

The FairCom dynamic dump provides a variety of options. These options are included in a script file. This section describes the script file and lists the script keywords and arguments available for defining a dynamic dump.

Backup Script File

The dump script file is a plain text file that specifies your options. For example, when to perform a dump, what interval to repeat a dump, and what files to include in a dump.

Format

The script file consists of a series of instructions, each of which is given by a keyword followed by a space and an argument (e.g., the keyword !DAY followed by the argument “WED”). All script keywords begin with an ‘!’ and are not case sensitive (i.e., !DAY = !Day). Arguments are strings of letters, numbers, and punctuation, in the format shown below for each keyword (e.g., WED). New lines divide script keywords and arguments. Keep each keyword/argument pair on a separate line, as in the sample script shown after the list of keywords.

With the following two exceptions, the order of keywords does not matter:

The next to last script keyword must be !FILES, followed by an argument which is a list of the files to be dumped one file name per line. Do NOT include a file name on the same line after the !FILES keyword.

The last script keyword in the script file must be !END, which takes no argument.

Example

!TIME 23:00:00

!DAY Sun

!DUMP SYSTEM.BAK

!DELAY 600

!FREQ 168

!FILES

FAIRCOM.FCS

!END

This script schedules a weekly dump, at 11:00 PM on Sunday nights. The only file included in the dump is FAIRCOM.FCS. The system will wait until 11:10 PM (i.e., 600 seconds delay, after the starting time of 11:00 PM) for all active transactions to complete and then it will abort any transactions still active and begin the actual dump. The dump data will be saved in the file named SYSTEM.BAK.

Note: The FairCom DB Server configuration file can also control the way lingering transactions are aborted.

Note: The server opens the dynamic dump script when the dump is scheduled, reads its contents, then closes it. At dump execution time the server opens the script again, reads the contents and then closes it before proceeding to dump the files.

!ALLOW_TRAN

The v13 revision adds support for creating dynamic dumps without a quiet checkpoint requirement.

Prior to v13, dynamic dumps required that all new transactions be stopped until outstanding transactions completed, allowing a checkpoint with no active transactions to be written to the transaction log. But this could cause an arbitrarily long delay if large transactions were active. The quiet checkpoint was required in order to have a valid starting point for a forward roll.

To request a backup without stopping all new transactions, add !ALLOW_TRAN to your dynamic dump script. This eliminates the delay caused by waiting for active transactions to complete, but causes the backup to be larger and additional processing to occur when restoring from the backup. This added overhead is proportional to the size of the active transactions at the time of the backup.

!ALLOW_TRAN eliminates the need for a quiet checkpoint during dynamic dump, unless server configuration option PREIMAGE_DUMP is active. When PREIMAGE_DUMP is active, !ALLOW_TRAN is ignored. When !ALLOW_TRAN is in effect, it modifies the behavior of the existing !DELAY <interval> dynamic dump option: if !DELAY is non-zero, block new transactions for up to <interval> seconds, waiting for a quiet transaction state.

If a quiet transaction state is reached, we create a regular backup. If <interval> expires and active transactions remain, continue the backup with active transactions. This allows the use of a short delay to possibly get a smaller and faster restoring backup if large transactions are unusual.

When a backup created with !ALLOW_TRAN is restored, any changes made by transactions that were active at the beginning point of the backup will be rolled back. A forward roll that begins from such a backup will apply these earlier transactions.

!BLOCK_RETURN

Forces ctdump to wait until the dynamic dump is completed before returning. Without BLOCK_RETURN, ctdump returns as soon as the FairCom Server receives the dump request. By waiting for completion, BLOCK_RETURN permits the System Administrator to determine when the dump is completed. Developers may find it useful to alert a System Administrator when a dynamic dump is complete.

!COMMENT

Default: Off

Informs the FairCom Server that the remainder of the script file is for documentation purposes only and is not to be evaluated. Do not place keywords after this keyword.

!COPY_NONCTREE

To include non c-tree files in a dynamic dump, use the dump keyword !COPY_NONCTREE. Any file included in the !FILES section of the FairCom Server dynamic dump script that receives error FUNK_ERR (13) or error READ_ERR (36) on c-tree open will be treated as a non c-tree file and copied directly to the dump stream base directory. More accurately, to the dump stream base directory plus any subdirectory included in the file's name.

If the destination directory does not exist, the FairCom Server will attempt to create it. If this directory creation fails a FCPY_ERR (796) is reported.

Note: A check is not made that wildcard specifications in the c-tree/non-ctree file sections match the same filename. In this case, the c-tree file is included in the dump and then the non-ctree file is also copied.

!DATE <mm/dd/yyyy>

Date to perform the dynamic dump or rollback. If the date has already passed, the !FREQ interval is applied to find the next scheduled time. If no !DATE or !DAY is specified, today is assumed.

!DAY <day of week>

Instead of a date, a day-of-week may be used to schedule the dump. They are specified as SUN, MON, TUE, WED, THR, FRI, or SAT. If no date, time or day-of-week is specified, then the dump is scheduled for immediate execution.

!DELAY <seconds>

Number of seconds to wait until aborting active transactions.

If zero, the FairCom Server will not abort active transactions. The dump waits indefinitely until all active transactions have completed and no new transactions will be permitted to begin.

If the delay value is greater than zero, the FairCom Server waits until either the delay has expired or all active transactions have completed. At this point, it begins the dynamic dump and permits new transactions to start up. If all transactions have not completed, the FairCom Server aborts those transactions still in progress, with one of two error messages:

TABN_ERR (78), indicates the transaction has been abandoned.
SGON_ERR (162), a generic error indicating a break in communication between the FairCom Server and the application.

Note: The default behavior is to wait for all pending transactions to complete. This may cause a timeout status (156 NTIM_ERR) to be logged to inform the DBA of this unexpected situation. That error does not imply anything about the dump, other than it waited for transaction to complete. If the DBA prefers to abort all pending transactions and start the dynamic dump immediately, the !DELAY script option can be used to abort the pending transactions after the specified number of seconds.

Option for Abandoning Dump

When the Dynamic Dump begins, by default it prevents new transactions from beginning and waits indefinitely for open transactions to complete. Once no transactions are active, a dump checkpoint is created, the dump creation begins, and transactions are allowed to resume. If the dump script option !DELAY <N> is specified, the Dynamic Dump waits up to N seconds for active transactions to complete before terminating them. This revision allows !DELAY <N> with N < 0 to cause the dump be abandoned with error DUMP_ABANDONED_ERR (1162) if active transactions still remain after |N| seconds.

!DUMP <dump file>

The name of the file or device into which all the data for all the dump files will be stored.

Note: If a file by this name already exists, it will be deleted at the beginning of the dump and the new dump file replaces the old file.

Note: There must be sufficient space for the dump file, which is limited to the maximum file size for the operating system (2 GB on some systems). If enough space is not available, the dump fails. A failure due to insufficient disk space will not corrupt anything, but additional space must be allocated before a dynamic dump is completed.

!END

Terminates the instructions in the script file. Place !END immediately after the !FILES keyword and list of files. !END takes no argument.

!ENDSEGMENT

Terminates the list of segments when specifying individual segment size and location.

!EXT_SIZE <bytes | NO>

Change the default extent (segment) size from 1GB or disable with NO. See Section Dump To Multiple Files - No Size Limit (Backup as Multiple Extent Files, Backup Extent Sizes) for more details.

!FILES

The !FILES keyword is followed by names of files to include in the dynamic dump or rollback. This must be the next to last keyword in the script file and it takes no arguments.

Filenames must begin following the !FILES keyword line with one line for each file. File names should not be listed on the same line as the !FILES keyword. The !END keyword terminates the list of files on a single line.

We strongly suggest that FAIRCOM.FCS be included in your list.

Members of a superfile cannot be individually “dumped.” The entire superfile must be dumped; that is, the name of the host superfile, not a superfile member, is placed in the list of files.

The * and ? wildcards are supported.

See !RECURSE for other options.

Wildcard Support for File Names

Dynamic dump backup and restore scripts specify the names of FairCom DB data and index files that are to be backed up or restored, delimited by the !FILES and !END script keywords. It is possible to specify filenames using the typical asterisk (*) and question mark (?) wildcard symbols. See FairCom DB Standard Wildcards.

In addition, the dynamic dump script keyword !RECURSE controls directory recursion when using wildcards. The !RECURSE keyword only applies when processing a !FILES entry containing a wildcard. Keep in mind that it is possible to specify standard file names and wildcard file names, one after the other, in the script. For example:

!RECURSE YES

!FILES

myfile.dat

cust*.dat

thedata.dat

*.idx

!END

There are three parameters for the !RECURSE keyword:

!RECURSE NO	Do not recurse subdirectories.
!RECURSE YES	Recurse underlying directories (max depth 32).
!RECURSE MATCH_SUBDIR	File names and directory names must match.

In the case of MATCH_SUBDIR, not only does the file name require a match on the wildcard pattern, but only directory names which match the pattern will be considered for recursion.

The dynamic dump is specifically designed to address FairCom DB data and index files, including superfiles. Please keep in mind: it is possible for your wildcard representation to represent non-c-tree files (see Non-ctree Files Included in a Dynamic Dump (Back Up Non-ctree Files, Back Up Non-ctree Files)). The following definitions cause all files within the server’s LOCAL_DIRECTORY to be considered. If any non-ctree files are encountered, the dynamic dump rejects them and a message is written to the CTSTATUS.FCS file if the DIAGNOSTICS DYNDUMP_LOG keyword is active. A rejection does NOT cause the dump to terminate. It will proceed to the next file.

!FILES *.* !FILES *

!END !END

Please remember that the dynamic dump does not support individual superfile member names. Specify the host superfile name in the script to back up the members. Here are examples of wildcard names:

the pattern m*t matches mark.dat but does not match mark.dtx
the pattern *dat matches markdat and mark.dat
the pattern *.dat only matches mark.dat (not markdat)

FairCom DB Files to NOT Include in a Backup

Certain c-tree housekeeping files should not be included in your dynamic dumps. When restoring dumps with these files in them, you may find you end up with DCRE_ERR (444) errors as these files collide with housekeeping files of the restore process itself. The following files should be excluded from your list of files to back up:

L*.FCS (Transaction Log files)
I*.FCS
S0000000.FCS (Transaction start file)
S0000001.FCS (Transaction start file)
D*.FCS

WARNING: Don't use *.FCS in your file list.

Exceptions:

FAIRCOM.FCS - Maintains user and group security information. ALWAYS back up this file.

SEQUENCEDT.FCS - Sequence number pool and index. If using the sequence number feature this file is a must to back up.

SEQUENCEIX.FCS

SYSLOGDT.FCS - System logs. If using this feature, consider backing up this file.

SYSLOGIX.FCS

!FREQ <hours>

Hours between successive dumps. For example, 24 to repeat the dump once a day, or 168 to repeat the dump once a week.

!FULL_PATH

The !FULL_PATH option ensures consistent naming for the files that are included in the backup.

When this option is specified, each file in the backup is listed by its full path.

When this option is not specified, the file name that is listed in the backup depends on whether or not the file is already open when the dump occurs:

A. If the file is already open, the path specification that was used to open the file, which might be a relative or full path, is listed in the dump.

B. If the file is not open, the path specified in the dump script is listed in the dump.

!IMMEDIATE_RESTORE

The FairCom Server dynamic dump script file keyword, !IMMEDIATE_RESTORE instructs the dynamic dump to be immediately followed by a dump restore. This allows transaction-consistent files to be available immediately in a native file system form as opposed to embedded files in the dynamic dump stream file.

The natural solution is to run ctrdmp in the directory that receives the dump stream file, which is called the dump stream base directory. In essence, this requires that !DUMP <streamFileSpec> use a file name including a path where the dump restore should run. For example, a dynamic dump script entry of the form

!DUMP I:\dump\mydumpstream

will cause the dump stream file mydumpstream to be created in the dump stream base directory I:\dump. If !IMMEDIATE_RESTORE is part of the dump script, then the automatically launched ctrdmp is also executed in the I:\dump directory.

It is recommended to launch ctrdmp.exe from a batch file called ctrdmp, which can reside in the server directory. The executable can reside elsewhere (i.e., in the dump stream base directory) and the batch file can call it using a path. The batch file can also do cleanup before (and after) a restore takes place, such as archiving from a prior restore.

Upon restoration of files, the enhanced dump restore will also automatically create any required directory hierarchies for previously backed up files. If an immediate restore operation fails, the server sets the error code for the dynamic dump operation to DRST_ERR (797, immediate dump restore failed).

!IMMEDIATE_RESTORE_LOGFILE

!IMMEDIATE_RESTORE_LOGFILE can be specified in a dynamic dump script to cause c-tree Server to write the output of the ctrdmp utility during the immediate restore to the specified file. This can be useful for troubleshooting failed immediate restores. For example, to write immediate restore output to the file ctrdmp.log in c-tree Server's working directory (in the LOCAL_DIRECTORY directory if that option is used), use this dump script option:

!IMMEDIATE_RESTORE_LOGFILE ctrdmp.log

Note that !IMMEDIATE_RESTORE_LOGFILE is only used a FairCom DB Server. It is ignored by the ctrdmp utility.

!IMMEDIATE_RESTORE_PROGRESS

Reads the output of the dump restore utility during immediate restore and uses the output to send progress notifications to the client that invoked the dump, if the client requested receiving dynamic dump progress notifications. The dump restore utility has been updated to output special messages that c-tree Server uses to identify progress as files are restored from the dynamic dump. In order to use the notification feature during an immediate restore of a dynamic dump, follow these steps:

Use a version of ctrdmp that supports the !IMMEDIATE_RESTORE_WITH_PROGRESS option into a directory that is in a FairCom DB server path.

Use the dump script option !IMMEDIATE_RESTORE_WITH_PROGRESS in the dump script rather than the !IMMEDIATE_RESTORE dump script option.

!ISAM

The !ISAM option provides a way to conveniently back up ISAM file sets and partitioned file sets.

When specified, each data file with an IFIL resource is opened at the ISAM level, and all related files are included in the backup. This includes the associated index files and partition member files.

!NONCTREEFILES

A dynamic dump script also supports listing specific files to be backed up as non-ctree files. If the !FILES list contains the !NONCTREEFILES keyword, all files following that keyword are treated as non c-tree files. Wildcard specifications are allowed. The !NONCTREEFILES keyword must appear after the !FILES keyword.

Also see the alternative method !COPY_NONCTREE

Note: The !NONCTREEFILES script keyword does not require specifying the !COPY_NONCTREE option in the script.

!PROTECT and !PROTECT_LOW

The keyword !PROTECT, without an argument, added to a dynamic dump script file suspends updates to each non-transaction file while it is being dumped. This ensures the file’s data integrity. The associated index files for a data file are not guaranteed to be consistent with the data file because the files are not dumped at the same time. With transaction files, the files are consistent because transaction log entries are used to bring all files back to the same point in time, i.e., the effective dump time. In most situations it is more efficient to dump only the data files and rebuild to recreate the indexes.

The update suspension is enforced only at the ISAM level unless the keyword !PROTECT_LOW is used instead. !PROTECT and !PROTECT_LOW are mutually exclusive options. The last one in the script is used. FairCom suggests using the !PROTECT_LOW when using low-level function calls.

Whether or not !PROTECT or !PROTECT_LOW are used, resource updates are suspended at the AddResource(), DeleteResource(), and UpdateResource() entry points.

!RECURSE <YES | NO | MATCH_SUBDIR>

Default is NO. Controls directory recursion when using wildcards. The !RECURSE keyword only applies when processing a !FILES entry containing a wildcard. In the case of MATCH_SUBDIR, not only does the file name require a match on the wildcard pattern, but only directory names which match the pattern will be considered for recursion.

!SEGMENT

See details in Segmented Dynamic Dump (Back Up as Segmented Files, Back Up Segmented Files).

!TIME <hh:mm:ss>

Time of day, on a 24-hour clock, to perform the dynamic dump or rollback. If the time has already passed, then the !FREQ interval is used to find the next scheduled time. If a !DATE or !DAY is specified without a time, then the time defaults to 23:59:59.

The script requires the use or leading zeros for the hour, minute, and second so that each contains two digits. For example, the valid entry for 6:00 is:

!TIME 06:00:00

The following is not a valid entry (notice the single digit, "6", for hours):

!TIME 6:00:00

If no time, day, or date is specified the dump begins immediately.

FairCom DB Files to Include for a Successful Backup Strategy

A FairCom DB SQL dictionary is composed of several files. You will need to back up all of these files if you want to be able to restore the entire SQL dictionary from your backup. By backing up the correct set of files, you will be able to do a full restore and have your SQL dictionary ready-to-go.

The following files need to be backed up if you want to be able to restore the entire SQL dictionary:

FAIRCOM.FCS
ctdbdict.fsd
*.dat in the ctreeSQL.dbs folder
*.idx in the ctreeSQL.dbs folder
ctreeSQL.fdd in ctreeSQL.dbs\SQL_SYS

The !FILES (!FILES, !FILES) section of your dynamic dump script will look like this:

!FILES

FAIRCOM.FCS

ctdbdict.fsd

ctreeSQL.dbs\*.dat

ctreeSQL.dbs\*.idx

ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd

!END

More generally, the following files are FairCom internal files that need to be included in backups to allow recovery to function without SKIP_MISSING_FILES YES (in the event these files are changed during the backup interval):

FAIRCOM.FCS
SYSLOG*.FCS
SEQUENCE*.FCS
DFRKSTATE*.FCS
ctdbdict.fsd
*.dbs\SQL_SYS\*.fdd
RSTPNT*.FCS
REPLSTATE*.FCS (created on the target server by the Replication Agent)

Testing the Backup

The following test should demonstrate that you have backed up everything you need:

Use the dynamic dump utility, ctdump (ctdump - Schedule Backup Utility, Submit Backup Utility), to back up your files into SYSTEM.BAK.

The !FILES (!FILES, !FILES) section of your dynamic dump script should include the entries shown earlier.

Shut down your FairCom Server and rename your C:\<faircom>\server\data folder to a new (unused) folder name, such as data.old:

C:\<faircom>\server\data.old

Create a new data folder and copy the following files to this location:

ctrdmp.exe

SYSTEM.BAK

Your backup script (the text file that contains the !FILES section shown above)

Run ctrdmp to restore your files in place.
Now start your FairCom Server and connect using FairCom DB Explorer. You should be able to see your restored SQL tables.

FairCom DB Files to NOT Include in a Backup

L*.FCS (Transaction Log files)
I*.FCS
S0000000.FCS (Transaction start file)
S0000001.FCS (Transaction start file)
D*.FCS

WARNING: Don't use *.FCS in your file list.

Exceptions:

FAIRCOM.FCS - Maintains user and group security information. ALWAYS back up this file.

SEQUENCEDT.FCS - Sequence number pool and index. If using the sequence number feature this file is a must to back up.

SEQUENCEIX.FCS

SYSLOGDT.FCS - System logs. If using this feature, consider backing up this file.

SYSLOGIX.FCS

Defer Backup I/O for Improved Performance

When a dynamic dump runs, the disk read and write operations of the backup process can slow the performance of other database operations. FairCom DB supports an option that allows an administrator to reduce the performance impact of the dynamic dump.

The FairCom DB configuration option:

DYNAMIC_DUMP_DEFER <milliseconds>

This option sets a time in milliseconds that the dynamic dump thread will sleep after each write of a 64KB block of data to the dump backup file.

An application developer can also use the c-tree ctSETCFG() API function to set the DYNAMIC_DUMP_DEFER value. For example, the following call specifies a 10-millisecond DYNAMIC_DUMP_DEFER time:

ctSETCFG( setcfgDYNAMIC_DUMP_DEFER, "10" );

The DYNAMIC_DUMP_DEFER value set by a call to ctSETCFG() takes effect immediately, so this API call can be used by administrators to adjust the speed of a running dynamic dump depending on the amount of other database activity.

Note: The maximum allowed DYNAMIC_DUMP_DEFER time is 5000 milliseconds, set at compile-time. If a value is specified that exceeds this limit, the DYNAMIC_DUMP_DEFER time is set to DYNAMIC_DUMP_DEFER_MAX.

The FairCom DB Administrator utility, ctadmn, was also updated to support the dump sleep time option to change this value at run time. The "Change Server Settings" menu is available from the main menu of the ctadmn utility.

Defer Interval

The DYNAMIC_DUMP_DEFER option causes the dynamic dump to pause for the specified number of milliseconds each time it writes 64 KB of data to the dynamic dump stream file. For large backups, even the smallest DYNAMIC_DUMP_DEFER value of 1 millisecond adds significant time to the dynamic dump. For example, 100 GB = 1600000 * 1 ms. = 1600 seconds of additional time.

An additional keyword, DYNAMIC_DUMP_DEFER_INTERVAL, specifies the number of 64 KB blocks that are written before the DYNAMIC_DUMP_DEFER sleep is performed. For example, DYNAMIC_DUMP_DEFER_INTERVAL 16 would cause the DYNAMIC_DUMP_DEFER sleep to occur after every 64 KB * 16 = 1 MB of data written to the dump stream file.

Note: If a value greater than 5000 is specified for DYNAMIC_DUMP_DEFER_INTERVAL, the value is set to 5000. If a value less than 1 is specified, the value is set to 1.

This option can be set by the ctSETCFG() API function:

ctSETCFG( setcfgDYNAMIC_DUMP_DEFER_INTERVAL, "16" );

A new menu option to set this value has been added to option 10 of the FairCom DB Server Administration (ctadmn) menu.

Back Up Non-ctree Files

Two alternative methods are available in the FairCom Server dynamic dump feature to allow ANY file to be backed up.

Specifying non-ctree Files

Non c-tree Files Dynamic Dump Script Example

!DUMP backup.dmp

!FILES

*.dat

*.idx

!NONCTREEFILES

*.log

*.txt

*.cfg

!END

Alternative Method

If the destination directory does not exist, the FairCom Server will attempt to create it. If this directory creation fails a FCPY_ERR (796) is reported.

Non-ctree File Keywords

!NONCTREEFILES

!COPY_NONCTREE

Note: The !NONCTREEFILES script keyword does not require specifying the !COPY_NONCTREE option in the script.

Back Up Files Without Transaction Control

It is possible to back up data files that are not under transaction control while the FairCom Server remains running. Of course, the safest way to perform a complete backup of data and index files while the FairCom Server remains running is to ensure that all your files are under transaction control. This way you are sure that all data and index files are completely synchronized, and updates to the files can continue during a dynamic dump.

Some developers choose not to implement transaction control for one reason or another. In some cases, developers migrating from the FairCom DB Standalone Multi-user model, FPUTFGET, to the FairCom Server, choose to use the FairCom Server in an FPUTFGET-like manner. An FPUTFGET-like server is defined with the following FairCom Server keywords:

COMPATIBILITY FORCE_WRITETHRU

COMPATIBILITY WTHRU_UPDFLG

Although it is possible to define a non-transaction controlled file within a dynamic dump backup script, there is no protection against updates to this file. In other words, it is possible for the file to be updated during the dynamic dump. Updating a file controlled by transaction processing is okay, because the dump restore process can use the transaction logs to restore to a consistent state. However, if files NOT under transaction control are updated while they are being backed up they cannot be backed up in a consistent state.

The keyword !PROTECT, without an argument, when added to a dynamic dump script file causes the non-transaction files to be dumped cleanly by suspending any updates while each file is dumped. At this point, the associated index files for a data file are not guaranteed to be consistent with the data file because the files are not dumped at the same time. Updates are only suspended while the data file is being backed up.

This technique ensures the data file is backed up in a known state. The restore process for a non-transaction control file MUST be complemented with an index rebuild. Because protection is for data files only, under most situations, the indexes are not worth dumping since they must be rebuilt.

Note: !PROTECT suspends updates at the ISAM level only. The keyword !PROTECT_LOW also suspends low-level updates in addition to the ISAM level. FairCom suggests using the !PROTECT_LOW when using low-level function calls.

Automatically Restore Backup for Ready-to-Use Files

The FairCom Server dynamic dump script file keyword, !IMMEDIATE_RESTORE, instructs the dynamic dump to be immediately followed by a dump restore. The idea is to allow for transaction consistent files to be available immediately in a native file system form as opposed to embedded files in the dynamic dump stream file.

A key issue is where the dynamic dump restore utility, ctrdmp, can run as it cannot run in the current server directory. If this occurs, error TCOL_ERR (537) results indicating that ctrdmp conflicted with an existing server operation. The natural solution is to run the ctrdmp in the directory that receives the dump stream file. We call this the dump stream base directory. In essence, this requires that !DUMP <streamFileSpec> use a file name including a path where the dump restore should run. For example, a dynamic dump script entry of the form

!DUMP I:\dump\mydumpstream

Upon restoration of files, the enhanced dump restore will also automatically create any required directory hierarchies for previously backed-up files. If an immediate restore operation fails, the server sets the error code for the dynamic dump operation to DRST_ERR (797, immediate dump restore failed).

Backup as Multiple Extent Files

The Dynamic Dump backup feature defaults to breaking-up the backup file (stream file) into multiple physical files (segments). This gets around individual file size limits imposed by the host OS (e.g., 2GB for a typical Unix system). Each backup file segment defaults to 1GB. There is no limit on the number of backup files (segments) supported.

Use the !EXT_SIZE keyword to change the segment size at runtime (up to 2000MB) by setting the argument of !EXT_SIZE to the desired number of bytes. Set the argument to NO to disable this feature and limit the dump to one file up to the OS maximum file size.

When a backup stream file is broken into segments, they are named as follows: original.001, original.002, etc, unless the original dump file has a name of the form name.nnn where nnn represent digits. For example, if the original dump file is named dump.str, the first additional segment after dump.str gets to the extent size will be dump.str.001. However, if the original dump file is named dump.111, then the first extent will be dump.112.

On some systems, the dynamic dump extent names formed from the original dump stream file name by adding .001, .002, etc. are not legal. Therefore, the extent name is first checked internally. If it does not work, the original dump stream file name is modified to produce a safe name in one of the following ways:

Replace name extension, if any, in original with numeric name extensions (.001, .002, etc.).
If the original name had no name extension, truncate the name to 8 bytes, and add numeric name extensions.
If the original name had no name extensions and is not more than 8 bytes, use the name FCSDDEXT.001 for the first dump stream segment, incrementing the numeric name extension as needed.

Back Up as Segmented Files

The FairCom Server and the ctdump and ctrdmp utilities support dynamic dumping of segmented files and the creation of segmented (stream) dump files.

Segmented dump files are different from the !EXT_SIZE feature that automatically breaks the dump file into 1GB ‘extents’. Dumping to segmented files allows you to take advantage of huge file support and to specify the files size and location for each dump file segment.

To dump segmented data or index files, simply list the host (main) file in the !FILES list and the segments will be managed automatically.
To cause the output dump file produced by the dynamic dump itself to be segmented, use these script entries:

!SEGMENT <size of host dump file in MB>

...

!ENDSEGMENT

The host dump file is the file specified in the usual !DUMP entry. Only the last segment in the !SEGMENT / !ENDSEGMENT list can have a zero size specified, which means unlimited size.

For example, assume bigdata.dat is a segmented file with segment names bigdata.sg1, bigdata.sg2, and bigdata.sg3, and the index file bigdata.idx is not segmented. To dump these files into a segmented dump file, use the script:

!DUMPd:\bigdump.hst

!SEGMENT 50

e:\bigdump.sg1 75

f:\bigdump.sg2 0

!ENDSEGMENT

!FILES

bigdata.dat

bigdata.idx

!END

The host dump file is up to 50 MB on volume D:, the first dump file segment is up to 75 MB on volume E:, and the last dump file segment is as large as necessary on volume F:.

Mirrored File Backups

Mirrored files are supported during dynamic dump and dump recovery as follows:

If a mirrored file should be opened for use by an application during a dynamic dump, the dump script should contain the “mirrored” name, i.e., the name with the vertical bar (‘|’). For example, sales.dat|msales.dat.
If this is not done, and the dynamic dump opens the primary file, because it is not in use, a client opening the primary|mirror combination gets an MNOT_ERR (551, file already opened without mirror). To avoid blocking users from gaining file access, open primary files with their mirrors when specified for dynamic dumps.
The dump recovery program recreates both the primary and mirror files. It reproduces the primary file, and copies it over the mirror file.

Backup Progress Messages Displayed in Function Monitor

During backup testing, watching the progress of a running dynamic dump can be beneficial. Adding the keyword DIAGNOSTICS DYNDUMP_LOG writes low-level progress messages to the CTSTATUS.FCS file. If the FUNCTION_MONITOR YES keyword is also active, dynamic dump progress information will also be written to the function monitor.

Mask Routine Backup Messages in CTSTATUS.FCS

Normally, a dynamic dump writes the names of all the files it backs up to the FairCom Server status log, CTSTATUS.FCS. The FairCom DB configuration option:

CTSTATUS_MASK DYNAMIC_DUMP_FILES

can be used to suppress the logging of the names of the files backed up by a dynamic dump operation. This option reduces the amount of information logged in CTSTATUS.FCS for easier analysis by an administrator.

Run Time Configuration

The ctSETCFG() function can be used to dynamically turn this option on or off while FairCom DB is running.

Examples

To turn the option on:

ctSETCFG("setcfgCTSTATUS_MASK", "DYNAMIC_DUMP_FILES");

To turn the option off:

ctSETCFG("setcfgCTSTATUS_MASK", "~DYNAMIC_DUMP_FILES");

Killing a Running Backup

To kill a dynamic dump, simply execute ctadmn and list active clients. The dynamic dump will appear with the COMM PROTOCOL set to DYNAMIC DUMP. Now use the kill clients option to terminate the process. This allows a backup procedure to be canceled (killed) after it has been submitted to the FairCom Server.

Database Backup

DYNAMIC_DUMP_DEFER

Allows an administrator to reduce the performance impact of the dynamic dump.

DYNAMIC_DUMP_DEFER_INTERVAL

Specifies the number of 64 KB blocks that are written before the DYNAMIC_DUMP_DEFER sleep is performed.

PERMIT_NONTRAN_DUMP

Improves backup performance by skipping non-transaction files and pre-image files during a dynamic dump.

PREIMAGE_DUMP

Causes all PREIMG files, even those not in a dynamic dump, to be temporarily changed to TRNLOG files and all transactions to be changed from PREIMG mode to TRNLOG mode during the dump.

VSS_WRITER

When enabled, FairCom DB loads the Volume Shadow Copy Service (VSS) writer DLL (c‑treeACEVSSWriter.dll) and initializes the VSS writer when the server starts.

Diagnostics

Diagnostics keywords are intended to be used to help collect data to be analyzed for troubleshooting. The use of these keywords should be avoided in normal operation as they may cause additional overhead (e.g., logging).

DIAGNOSTICS DYNDUMP_LOG

Generates dynamic dump status info, sending progress entries during each dynamic dump to CTSTATUS.FCS.

DIAGNOSTICS VSS_WRITER

Enables VSS writer to log diagnostic messages to CTSTATUS.FCS.

Database Backup Guide

Scheduling a Database Backup

ctdump - Schedule Backup Utility

Backup Script Options

Backup Script File

!ALLOW_TRAN

!BLOCK_RETURN

!COMMENT

!COPY_NONCTREE

!DATE <mm/dd/yyyy>

!DAY <day of week>

!DELAY <seconds>

!DUMP <dump file>

!END

!ENDSEGMENT

!EXT_SIZE <bytes | NO>

!FILES

Wildcard Support for File Names

FairCom DB Files to NOT Include in a Backup

!FREQ <hours>

!FULL_PATH

!IMMEDIATE_RESTORE

!IMMEDIATE_RESTORE_LOGFILE

!IMMEDIATE_RESTORE_PROGRESS

!ISAM

!NONCTREEFILES

!PROTECT and !PROTECT_LOW

!RECURSE <YES | NO | MATCH_SUBDIR>

!SEGMENT

!TIME <hh:mm:ss>

FairCom DB Files to Include for a Successful Backup Strategy

FairCom DB Files to NOT Include in a Backup

Defer Backup I/O for Improved Performance

Back Up Non-ctree Files

Back Up Files Without Transaction Control

Automatically Restore Backup for Ready-to-Use Files

Backup as Multiple Extent Files

Back Up as Segmented Files

Mirrored File Backups

Backup Progress Messages Displayed in Function Monitor

Mask Routine Backup Messages in CTSTATUS.FCS

Killing a Running Backup

Database Backup

Was this article helpful?

Related Articles

Configuring FairCom DB SQL Data Sources