Standalone Multi-User

There are significant differences in some aspects of FairCom DB between traditional multi-user mode without a server and client/server mode. This chapter discusses the specifics of multi-user mode when not using a FairCom Server.

With traditional multi-user mode the operating system has a great deal of influence on FairCom DB. As discussed in “Multi-User Concepts”, we use both write locks and read locks. However, not all operating systems supply both. In addition, the manner in which FairCom DB uses write locks may vary from the way that the operating system implements them. Finally, the operating system may limit the number of locks available to your FairCom DB application.

Two types of record locks must be distinguished in FairCom DB: record locks applied to the B-Tree nodes in the index file, and record locks applied to a program’s data records. While the reason for the locks is almost identical, avoiding simultaneous and conflicting updates to the data, there are significant differences in the management of these two classes of locks. FairCom DB automatically takes care of any required node locks, while your application program must control the data record locks.

Node Locks

In the non-server, multi-user model, when two programs attempt to update the same node, one must wait until the other has completed its update. The FairCom DB algorithm has all the necessary calls to the locking routines coded into place.

Because of the robust nature of the FairCom DB B-Tree algorithm, it is not necessary for a program that is merely searching (i.e., not updating) the index file to be locked out of any node, even if the node is currently being updated by another program. This leads to a special case: if only one “user” in a non-server, multi-user system is allowed to update the index files, then it is not necessary to actually place locks on nodes which are being updated.

Another important aspect of the special B-Tree algorithm used in FairCom DB is that competing node locks are guaranteed not to cause a deadlock situation. Therefore, the node lock routines can ignore the problems associated with deadlocks.

To use the FairCom DB index files in a non-server, multi-user system, the following steps must be taken:

• Set the disk protocol in ctoptn.h to FPUTFGET; and
• If more than one user is allowed to update the index structures at the same time, the node lock routines (LOCK() and UNLOCK()) in ctslck_a.c must be coded to access the operating systems locking protocols.

For most environments, these routines have already been implemented. If your operating system/compiler environment is one for which FairCom DB LOCK() and UNLOCK() have not been implemented, and if you intend to run in the Standalone Multi-user mode of operation, then examine the compiler’s run-time library documentation to determine how locks are implemented. Also, examine the implementation of locks in the ctslck_a.c module supplied for the Microsoft C compilers and the ctslck_a.c supplied for Unix system.

Note: The most significant feature of LOCK() is that it loops until it acquires a node lock. The looping is permitted since the index routines are guaranteed deadlock free.

Data Record Lock Implementations

For non-server, multi-user implementations, the same basic steps are required for the data record DLOCK(), RLOCK(), and UDLOCK() routines as are required for the node LOCK() and UNLOCK() routines described above. If your system is not already supported by an appropriate ctclib.c, keep in mind that data record locks are NOT guaranteed to be deadlock free. Therefore, where the (node) LOCK routine can wait for a lock, since they are guaranteed deadlock free, the DLOCK() and RLOCK() routines must return immediately if a lock conflict is encountered. Error DLOK_ERR (42) signifies that another process owns a conflicting lock on the same data record.

The following three defines control locking with FairCom Standalone Multi-user applications. The model best suited for each particular operating system is defined by FairCom at compile time. To override the default, place one of the following defines at the top of ctoptn.h or in the make file that produces the ctoptn.h:

1. ctLOCK_DIRECT - lock on actual record; used for most Unix and Apple Macintosh systems.
2. ctLOCK_EXTENDED - lock on region mapped to end of file; used for most non-Unix systems.
3. ctLOCK_DUMMY - lock on region mapped to dummy lock file; used for most non-Unix systems where extensive virtual files are required.

ctLOCK_EXTENDED or ctLOCK_DUMMY are used in operating system environments not permitting a region locked for update to be read by another process. (Early versions of Windows, for example.) The use of these techniques limits the maximum amount of data stored in each file. The limits for ctLOCK_EXTENDED are discussed below. If the limit is exceeded, the lock routine returns an error code of MAPL_ERR (653).

To revert to the dummy lock file style of locking, the default for non-Unix operating systems with previous FairCom DB releases, use the ctLOCK_DUMMY define.

When either ctLOCK_DIRECT or ctLOCK_EXTENDED are used, it is not permissible for a “virtual” file to be closed when locks are pending. The ctLOKCNT define, enabled by default when using these locking methods, causes FairCom DB to counts the number of locks on each file to avoid a virtual close when locks are pending. For the same basic reason, when you save a file with locks pending you cannot rely on closing and reopening the file since pending locks would be lost.

When using the ctLOCK_EXTENDED define as an override to the default selected by FairCom for your environment, be cautioned that the FairCom DB file size extension procedure closes and reopens a file on some systems when the xtdsiz parameter is non-zero. The file closing causes errors if locks were pending on the file. If you attempt to use ctLOCK_EXTENDED when file close/reopen is used during file extension, you should receive a compile error from the ctsave.c module.

Windows does not allow one process to read a record when another process has it locked. FairCom DB supports this capability by using Extended Locking, the current default, or a “Dummy Lock File.” The following table is a summary of the advantages and disadvantages of each method. Direct locking has all of these advantages, and none of the disadvantages.

Extended Lock Strategy

The Extended Lock Strategy locks a dummy, or extended, region of the data file uniquely related to the actual region. Our intent is not to lock the actual region, but to lock a uniquely related region that never has to be read. This is accomplished by adding a suitably large offset to the actual byte position of the lock. In this way, the lock is placed on the actual file involved, and no file numbering convention is required. The size of the offset must clearly exceed the largest possible actual record offset for your application.

Because the locks are held on the actual file, a file with pending locks cannot be virtually closed since the locks would be lost. This reduces the effectiveness of the Virtual File logic since not all virtual files will be available for closing.

The extended lock approach maps a lock request on an actual record to a corresponding byte at the end of the file, which cannot hold data. Based on a 2GB file size limit and our default values for ctMAX_USRLOKS, 256 the maximum of concurrent locks per each user process, and ctMAX_MBRLOKS, 1024 - the maximum number of locks which can be placed within a superfile, the extended lock approach yields the following data capacities per file:

Fixed-length data file or index file:

((2GB - (256 * 1024)) / (recSize + 1)  * recSize)

For example, if recSize is 256, the data capacity is about 1.99GB, where 1GB = 1024 * 1024 * 1024.

Variable-length or superfile:

(2GB - (256 * 1024)) * 10 / 11)

This computes to about 1.81GB of data capacity. In variable-length files or superfiles part of the file is occupied by record headers maintained transparently by FairCom DB.

Complications with Superfiles in Standalone Multi-user Models

To lock superfile members when the host is opened in shared mode requires a special case of the ctLOCK_EXTENDED technique. It is enabled with ctSUPER_MBRLOCK and may be applied to either ctLOCK_DIRECT or ctLOCK_EXTENDED. In prior releases ctSUPER_MBRLOCK was set in the FPUTFGET mode. Now, this option is inactive by default, and therefore now defaults to not supporting SuperFile Member file exclusive locks without modification. This improves performance, but prevents a Superfile member from being opened EXCLUSIVE when the superfile host is open SHARED. Compile with #define ctSUPER_MBRLOCK to allow applications to perform EXCLUSIVE opens of superfile members when the host file is open SHARED.

Complications with ctLOCK_TOP

Extended locking, the standard locking method for non-Unix systems in the FairCom DB Standalone Multi-user model, uses ctLOCK_TOP to determine the highest potential offset within a file.

Note: Extended locking will only work if all applications sharing files use the same offset, otherwise locks will not be properly implemented.

ctLOCK_TOP is not an issue with any of the FairCom DB client/server models, which automatically handle locking for any type of FairCom DB client.

When mixing platforms in the Standalone Multi-user model, ensure the ctLOCK_TOP define is consistent between different platforms. Use the lowest common setting to maintain compatibility. For example, when mixing 16-bit and 32-bit applications, use 0x7fffffff. The default settings for Standard libraries are:

When using FairCom DB Professional with ctHUGEFILE defined, there are additional limitations.

Dummy Lock File

The Standalone Multi-user mode of FairCom DB requires a dummy lock file or use of the Extended Lock Strategy, discussed above, when using environments not supporting robust write locks. For example, platforms only supporting exclusive record locks. No other process can read or write an exclusively locked record. FairCom DB does not require such strong locks. A write lock allows other processes to read the locked record, but not write it. To simulate write locks, FairCom DB secures true exclusive record locks on phantom regions of the dummy file. The dummy file is a separate FairCom DB data file used to help control record locking. Since the lock is held in a separate file, the virtual file logic does not affect locks held on the dummy lock file.

If two applications share one or more files in a multi-user FairCom DB environment using a dummy file, the following conditions must be met:

• They must use the same dummy lock file which must be numbered zero in the ISAM parameter file, or use the same IFIL structures with dfilno set to zero, or be assigned file number 0 if using low level functions.
• The dummy lock file should be opened before any other ctree files.
• Files shared between the applications must be assigned the same FairCom DB file numbers.
• Files not in common must not be assigned the same file numbers.
• There are artificial limitations on the quantity of concurrently open files and the maximum size of these files. In the “dummy lock file” approach, each region of data or index file is mapped to a unique byte of a single file on which the locks are actually requested. For example, an index node would be mapped to a byte in the dummy lock file based on the following equation:

node |= ((ULONG8) knum->filnum << LEFT_SHIFT);

Based on a 64 bit locking domain, the LEFT_SHIFT values yield the following constraints on quantity of concurrently opened files and the maximum file size:

In the above table, “256 TB records” means that a fixed-length data file can contain up to 256 * 1024 * 1024 * 1024 * 1024 records regardless of the record length. By comparison, the variable-length file limit is on the actual file size. The default LEFT_SHIFT setting is set as follows:

#define LEFT_SHIFT  49 /* Controls dummy lock capacity */

The file number dependency is because the file number is used in conjunction with the record offset to obtain a unique byte in the dummy file to lock. If the same file is opened with a different file number, the same offset would not be locked, preventing the logic from working. Additionally, all applications sharing a dummy lock file must use the same LEFT_SHIFT or the behavior is undefined.

To make it easier to assign file numbers consistent with the above three points, remember that you can inflate the fils parameter in InitISAM() or the idxs parameter of the parameter file to ensure sufficient flexibility in numbering the files. For example, if you call InitISAM() with fils set to ten (10), but you are only using two files, they can be assigned any file numbers between zero (0) and nine (9).

The dummy lock file must be assigned file number 0. This file is an optional file to enable write locks on Standalone Multi-user systems that only support exclusive locks.

NOTE: When used, a dummy lock file must always be opened as a permanent file, NOT a virtual file. Therefore, use a file mode of 3.

CAUTION: Windows OS file shares (NFS) rely on filesystem locks to maintain filesystem cache coherency. The use of a Dummy Lock File bypasses this mechanism. Updates to a shared file on NFS cannot be done safely using a Dummy Lock File. A different locking model is required to handle this safely.

Exclusive File Opens

Unix does not support effective exclusive file opens. If you remove #define ctNOEXCL, Unix supports exclusive file opens consistent with other environments, such as our client/server model or non-Unix FPUTFGET. However, this support uses a lock for each file opened by each user.

Defining ctLOCK_EXCLUSIVE when compiling ctclib.c undefines ctNOEXCL. This locks only the user lock region on an exclusive open. The disadvantage of this approach is that non-FairCom DB applications can update the file opened in exclusive mode within FairCom DB. To lock the entire file, use #define ctLOCK_EXCLUSIVE file. Locking the entire file appears to be very inefficient, especially with NFS mounted file systems, but it prevents non-FairCom DB applications from accessing the file.

To lock superfile members when the host is opened in shared mode requires a special case of the ctLOCK_EXTENDED technique. It is enabled with CTSUPER_MBRLOCK and may be applied to either ctLOCK_DIRECT or ctLOCK_EXTENDED.

Note: With exclusive file opens for Unix, and superfile member exclusive opens for both Unix and non-Unix systems, the maximum number of users supported is limited by ctMAX_USRLOKS, which defaults to 256. It may be overridden with your own #define ctMAX_USRLOKS placed in ctree.mak or ctoptn.h.

Exclusive file locks for superfile members, when the host is opened shared, are supported for ctLOCK_DIRECT and ctLOCK_EXTENDED unless CTSUPER_noMBRLOCK is defined. Disable this feature to reduce the number of locks held by an application. If this feature is not disabled, the maximum number of superfile members in a single host file is limited by ctMAX_MBRLOKS, which defaults to 1024, but may be overridden with your own #define ctMAX_MBRLOKS.

EXCLUSIVE File Caching

c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET) specifically does not cache data. Because it is designed to allow multiple users to manage the data simultaneously, all reads and writes to the database are forced (e.g., all reads are done directly from the disk and write operations flush (sync) all data to disk. The internal acronym FPUTFGET stands for “force put/force get”, or in other words, “force write” and “force-read”.

However, when a file is opened in EXCLUSIVE mode, there is no sharing of the file, and therefore no need to force reads and writes. Cache can be used to gain better performance.

When a file is opened in EXCLUSIVE mode in c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET), it can be cached. In order to take advantage of this feature, use either the InitCTreeXtd() or the InitISAMXtd() extended initialization call.

Note: If a non-extended initialization call is used (INTREE or INTISAM) under FPUTFGET, there will be NO data cache allocated..

As always, a data file that is NOT opened EXCLUSIVE will not be cached and therefore not use any of the data cache pages.

Note: If a cached file is not properly closed, its header will be marked with a “dirty” flag, and any subsequent open to this file will fail with a FCRP_ERR (14) error, indicating that the file’s data may not be intact.

Multi-User Non-Server Mode Rebuild

FairCom DB offers the capability to rebuild files in Standalone Multi-user mode. For improved efficiency FairCom still recommends all file rebuilds to be done in single-user mode whenever possible. For an example of rebuilding using RebuildIFile(), see the sample program ctixmg.c.

Tutorial for diagnosing invalid file headers for the Standalone Multi-User library

Diagnostics checks have been added to the Standalone Multi-User (FPUTFGET) library to help track down incorrect header values in the even that a data file is writing unexpected values to the header. The checks are controlled by setting the following environment variables before running a process that uses the Standalone Multi-User library.

1. To enable checking of the header fields when reading and writing the header, set the environment variable FAIRCOM_CHECK_HEADER_VALUES to "yes". For example: setenv FAIRCOM_CHECK_HEADER_VALUES yes
2. To enable diagnostic logging of cases where the first segment file descriptor is cleared when closing a file, set the environment variable FAIRCOM_CHECK_FILE_CLOSE to "yes". For example: setenv FAIRCOM_CHECK_FILE_CLOSE yes
3. When the diagnostic checks that are controlled by the FAIRCOM_CHECK_HEADER_VALUES and FAIRCOM_CHECK_FILE_CLOSE environment variables are enabled, and the environment variable FAIRCOM_PROCESS_DUMP_TYPE has been set to one of the following values prior to initializing the database engine, then when a diagnostic check fails, the library creates a process dump:
   • Setting FAIRCOM_PROCESS_DUMP_TYPE to "stack" enables creation of a process stack dump. For example:
   • Setting FAIRCOM_PROCESS_DUMP_TYPE to "full" enables creation of a full process dump. For example:

NOTE: If you have enabled process dump creation, be sure to install the required utility for your system (pstack, gcore, or procdump).