Stay up with the latest FairCom information. The following notifications have been found to be important considerations in many production systems. Please Contact your nearest FairCom office if you are concerned if any of these may impact your application or have any questions.

Post-V11.5 Updates

• V11.5 Delete Node Reuse List Default Behavior Change

Important V11.5 Updates

• V11.5 Notable Compatibility Changes
• V11.5 FairCom Server Critical Fixes
• V11.5 FairCom DB SQL Server Critical Fixes
• Linux Caching Considerations

V11 Issues

• V11 Critical Updates
• V11 Notable Compatibility Changes

 Pre-V11 Issues

• Highly Concurrent FairCom DB SQL Updates Involving Floating Point Conversions
• Prevent FPUTFGET Data Corruption with Concurrent Updates
• Potential Variable Length Data Corruption Prevented During Automatic Recovery
• Prevent Termination of FairCom DB from LRU Cache Miss Limitations
• Potential FairCom DB Automatic Recovery Failures with the LOGIDX Feature
• Avoid Index Errors Caused by memcpy() Implementation on Latest Operating Systems
• Correct Handling of Segmented Files During Automatic Recovery
• Microsoft Vista Locks Out FPUTFGET
• Transaction Number Rollover

Delete Node Reuse List Default Behavior Change

In V11.5 (and FairCom RTG V2.5) and later, deleted index nodes are not reused until after the file is closed. This can cause indexes to grow faster than expected.

A new server keyword NODE_REUSE_DELAY allows setting the node reuse time directly. This keyword sets the number of seconds after an empty index node is pruned from the index before it may be reused (default 0). Negative values disable index node reuse until the file has been closed and reopened. Prior to V11.5, this defaulted to 225 and was 0.75 * NODE_DELAY (minimum 4).

The deleted nodes are cleaned up when the files are closed, which means that this issue is a concern only if files are held open for long periods of time.

Pre-V11 Issues

Highly Concurrent FairCom DB SQL Updates Involving Floating Point Conversions

1 October 2011

Affected Builds: All FairCom DB SQL builds prior to 110914
Criteria: Highly concurrent FairCom DB SQL insertions involving floating point conversions 
Indications: Unexpected KDUP_ERR (2) errors

A potential data integrity issue was identified in FairCom's internal testing that demonstrated a potential field overwrite that could occur with highly concurrent FairCom DB SQL updates that include a floating point conversion. Affected field types include:

NUMERIC, NUMBER, DECIMAL, MONEY, (N)CHAR, (N)VARCHAR

A remote possibility exists with multi-threaded FairCom DB SQL applications (excluding JDBC and ADO.Net) when a floating point value (either a variable in the application using native floating points or a FLOAT SQL type) requires conversion into an aforementioned FairCom DB SQL type.

A combination of events must take place for this overwrite to occur:

1. Floating point data needs to be assigned to one of the above types,
2. Floating point data needs to be compared with one of the above types,
3. The execution of a SQL statement operation (or function) involving floating points and one of the above types.

Consider the case of an ODBC driver binding a MONEY column to a SQL_C_DOUBLE type. When an insert is performed, the C type data undergoes a conversion to an SQL type. During this conversion, there is a possibility a buffer is not protected from multi-threaded updates.

Solution: This possibility has been avoided by ensuring proper thread safe C library functions are used on respective platforms and the buffer used for conversion is properly protected.

FairCom customers on current maintenance can request an updated V9 FairCom DB SQL line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Prevent FPUTFGET Data Corruption with Concurrent Updates

20 January 2010

Affected Builds: All builds between 050722 through 100120 
Criteria: Concurrent FPUTFGET record add/updates to variable length files.
Indications: Unexpected ITIM_ERROR (160) errors

In c-tree multi-user standalone (FPUTFGET) mode, when a process updates a record, increasing the size of the record, while at the same time another process adds a record, and if there is deleted space available after the record that is being updated, the update logic could reuse that space even though the add operation has already used the space for the newly-added record.

This issue only affects the c-tree multiuser standalone mode (FPUTFGET) and does not affect the c-tree Server.

A code change applied in July 2005 modified previously correct behavior. As a result, the logic in RWTVREC() that reads the key from the space management index after locking the space that is a candidate for reuse skips the call to read the key from the space management index. This omission can cause the update to enlarge the record into space that has been reused by a record add operation. The ITIM_ERR (160) was observed in that an index key entry pointed to an offset in the data file that was within another record. That record had been enlarged by an update operation, and it overwrote a record that was added at the same time the record was updated. A simple rearrangement of code prevents this behavior.

FairCom customers on current maintenance can request an updated V9 FairCom DB line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Potential Variable Length Data Corruption Prevented During Automatic Recovery

5 October 2009

Affected Builds: All builds prior to 091005
Criteria: Variable Length Files under Transaction Control that have Undergone Automatic Recovery After a Failed Server Process
Indications: Error 37 during recovery and/or evidence of data or index corruption

Sometime after automatic recovery on a system where the FairCom DB Server had previously gone through the process of automatic recovery, data and index files were found to contain unexpected 10-byte headers beginning with 0xfbbf marks. The headers appeared at unexpected locations, sometimes overwriting data records or index nodes, and sometimes written at the physical end of the file, preceded by a region of 0x00 bytes.

A second observed symptom was automatic recovery failing with error 37 due to an invalid file descriptor, as indicated by the following entries in CTSTATUS.FCS:

- User# 00002 trandat: scanning log 815
Wed Sep 23 15:44:25 2009
- User# 00002 WRITE_ERR: ???? at 0:1457a4ex sysiocod=6 bufsiz=10 bytes written=0[0] ioLoc=0: 37

Automatic recovery invalidates the space management index for variable-length data files, and it queues entries for the space reclamation thread to process. These entries trigger the space reclamation thread to reconstruct the space management index by physically scanning the variable-length data files for deleted records and adding keys to the space management index for each deleted region found in the file.

Changes to the space management index can place entries - with associated file numbers - into the transaction log. One such entry was written to the server's preImage space, however, not immediately written to the transaction logs. A later transaction can then commit this entry with a transaction file number not associated with the original file, as the original file could have been physically closed and the file number reassigned to another physical file.

Should the server then experience an abnormal failure and automatic recovery takes place, the entry is processed from the transaction log with a transaction file number associated with the wrong physical file. An attempt is then made to write this entry to the incorrect file causing either data to be overwritten, or a failed write due to an invalid file descriptor.

To prevent this behavior, the transaction log entries are now written immediately and directly to the physical transaction log ensuring a correct associated transaction file number upon recovery.

FairCom customers on current maintenance can request an updated V9 FairCom DB line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Prevent Termination of FairCom DB from LRU Cache Miss Limitations

4 June 2009

Affected Builds:All V9 lines with build dates prior to 090602
Criteria: FairCom DB Server with DATA_LRU_LISTS or INDEX_LRU_LISTS configurations set greater than 1
Indications: Server crash after large number of cache misses

When a FairCom DB Server is running with the configuration options DATA_LRU_LISTS or INDEX_LRU_LISTS set greater than 1, after a large number (2^32, over 2 billion) data or index cache misses occur, the c-tree Server terminates with an unhandled exception. These options default to 4 in FairCom DB Version 9. Thus all FairCom DB Version 9 servers are susceptible to this situation. You can verify these options in the server startup information found in the server status log file, CTSTATUS.FCS.

When a cache page is required to hold a page that is not already in cache and multiple data or index LRU lists are in use, the data and index cache logic increments a variable used to calculate which data or index LRU list is to be used . However, the variable was declared as a signed long integer, and after 2^32 increments, became negative, resulting in a negative index offset for an array of data/index cache LRU list mutexes. Redeclaring the variables as unsigned long integers resolves this issue.

An immediate fix for anyone with potential to be affected by this Version 9.0 only issue is to directly specify DATA_LRU_LISTS 1 and INDEX_LRU_LISTS 1 in the FairCom DB configuration file (ctsrvr.cfg) to avoid this unhandled exception condition. You will need to restart your server to enable this configuration change.

FairCom customers on current maintenance can request an updated V9 FairCom DB line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Potential FairCom DB Automatic Recovery Failures with the LOGIDX Feature

22 May 2009

Affected Builds: All V8 and V9 lines that enabled this feature through the use of the LOGIDX file mode within the application or through the server configuration keyword FORCE_LOGIDX YES. This feature was enabled by default for FairCom DB V9 Servers with build dates 080111 through 090126.
Criteria: FORCE_LOGIDX YES enabled in the FairCom DB Server configuration
Indications: Failed recovery

FairCom DB provides a feature called LOGIDX which can allow for quicker automatic recovery time on server startup should you experience an unexpected failure (for example, a power outage). This feature can be enabled with a file mode of LOGIDX, or using the FORCE_LOGIDX server configuration keyword.

Extensive testing identified an isolated possibility of failure during automatic recovery when the LOGIDX logic is active. The net effect is a possibility for failed recovery in the event the c-tree Server was not properly shut down. This has been corrected as of V9.1 build 090522.

A simple solution to ensure you will not be impacted by this default setting is to disable the LOGIDX feature via your server configuration file. Follow these easy steps to perform this change:

1. Cleanly shut down your c-tree Server. (For example, use the c-tree Server Administrator utility, ctadmn, or the c-tree Server stop utility, ctstop.)
2. Add or change the following server configuration keyword in your server configuration file, ctsrvr.cfg:

FORCE_LOGIDX OFF

3. Restart your FairCom DB per your normal operating procedures.

Avoid Index Errors Caused by memcpy() Implementation on Latest Operating Systems

20 December 2011

The latest Linux versions, notably Redhat 6 (RHEL 6) have demonstrated a potential problem with FairCom DB buffer management functions. The most common symptoms are ITIM_ERR errors (160) or KDEL_ERR errors (4).

Previously, FairCom DB used a C library memcpy() function in certain buffer management areas. However, when the source and destination buffer locations overlap, this can cause unexpected memory corruption issues, and the latest Linux versions appear to exhibit this with greater frequency. The memmove() function has been utilized in place of the memcpy() function to avoid this problem. Latest versions of FairCom DB since V9.5 (builds since 110420) incorporate this change.

In most cases, an updated server in this environment, and a quick index rebuild is all that is needed. Existing data has been shown to be secure.

FairCom customers on current maintenance can request an updated V9 server line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Correct Handling of Segmented Files During Automatic Recovery

10 June 2009

Affected Builds: All builds prior to 090610
Criteria: Segmented file, Automatic recovery after a failed FairCom DB Server process
Indications: Renamed file segments

During automatic recovery of a failed FairCom DB process, existing file segments were renamed and thus subsequently not available to the application resulting in apparent missing data. During recovery, an attempt is made to open all segments of a segmented file found in the transaction logs. To open the segments, the segment definition resource must be read from the primary segment. However, an internal file map attribute had not been initialized resulting in a failed call. This failed call caused the recovery process to believe the additional segments did not exist. When later attempting to create the file segment, and that segment already existed, FairCom DB then renamed the segment. The uninitialized attributes are now properly assigned avoiding this potential incorrect renaming of segmented files.

FairCom customers on current maintenance can request an updated V9 server line at any time. Please contact your nearest FairCom office should you have any concerns that you are impacted by this update.

Microsoft Vista Locks Out FPUTFGET

Two independent issues affect FairCom's multi-user standalone operational model (FPUTFGET) which relies on the underlying operating system to ensure file locking integrity. This contrasts with FairCom's server-based models where locking is independently and solely maintained by the c-tree Server. These issues manifest in either of the following scenarios:

1. In environments where all machines accessing the data are using Windows Vista, it is possible for one c-tree Plus FPUTFGET application to “hang” when obtaining the record lock already owned by another FPUTFGET process. This impacts database files from many vendors and a patch from Microsoft is available.
2. In mixed OS environments (Windows XP with Vista for example) indexes can become out of sync resulting in ITIM_ERR (160) errors or even data corruption. This impacts versions of c-tree Plus V7 and V8 prior to V8.27 Build 071005 with HUGE files enabled, which is the default in V8.14 and later. This can also occur when using the c-tree Plus ODBC Driver in read/write mode and the FPUTFGET model of operation.

Applications using the c-tree Server and c-treeSQL Server are not impacted by these issues.

Testing your FPUTFGET Application

It is easy to test your c-tree Plus FPUTFGET application for this behavior. The c-tree Plus example program, ctixmg, provides an easy-to-use application allowing for a simple controlled test. This example is built when you include samples in your c-tree Plus mtree build.

1. Build ctixmg choosing the multiuser standalone (FPUTFGET) c-tree Plus model.
2. Start two ctixmg instances on two different machines. (Vista to Vista or Vista to XP for example)
3. B)egin Locking from the first instance of ctixmg.
4. A)dd a record from the first instance of ctixmg and do not E)nd Locking.
5. B)egin Locking on the second instance of ctixmg.
6. Attempt to U)pdate the same record as added in Step 4 from the second instance of ctixmg.

You should receive a DLOK_ERR error (42) from the second instance indicating that the expected and appropriate locking has taken place. If your application hangs (Vista to Vista), or the Update succeeds (Vista to XP), then you may require the Windows Vista hotfix and/or a c-tree Plus update.

What if I am affected?

All FairCom maintenance customers in good standing have been notified of this serious matter. A FairCom DB update is available. Now is also a great time to consider FairCom's FairCom DB technology which does not rely on the operating system to manage locking. With only a few simple changes, you can be up and running with enhanced speed, performance, huge data caching, dynamic backups, and the security of knowing your data is intact and secured!

Contact your nearest FairCom office should you have any questions or believe you are impacted by this potentially serious issue.

Transaction Number Rollover

Six-byte transaction numbers allow you to take advantage of a vast volume of available numbers to greatly increase the lifespan of your active FairCom DB data files.

Some transaction-intensive applications using older versions of FairCom DB have experienced exhausting these numbers. With faster and faster hardware, and greater volumes than ever, there are good reasons to migrate to the latest version of FairCom Server.

Consider a moderately high volume application with 1000 transactions per second:

Transaction Number Limits

• V6 supports around 1 billion transactions: 1000 tps for ~11.6 days.
• V8 supports 70 x 10¹²: 1000 tps for ~2000 years.

FairCom has created a comprehensive white paper describing this limitation of four byte transaction numbers, how to recover when you exhaust your supply of numbers, and how to convert and take advantage of six byte transaction number support. Download this valuable resource today! Six Byte Transaction Number Support

Contact FairCom and put the latest version of FairCom DB in action today!

Migrate to 6-Byte Transaction #’s in 5 Easy Steps

1. Relink all client applications with c-tree Plus V8 libraries.
2. Install a new c-tree V8 Server or newer using recommended practices.
3. Ensure ALL FairCom DB Transaction Logs (*.FCS) are removed.

4. Ensure all data files have an extended header which allows huge file and six byte transaction numbers. V6 c-tree Plus data files may be converted using the c-tree Plus Conversion utility, ctcv67. Be sure to observe the note below concerning this standalone application.
5. After starting your new FairCom DB, ensure ALL indexes are rebuilt using FairCom DB. Remember, existing indexes that have not had an appropriate extended header added will be rebuilt with the old four byte number support.

V12 Issues

Older .NET configurations detected correctly

The ADO.NET driver integration could not be used in Visual Studio because Visual Studio was unable to locate the correct driver to use. The following error could be seen: "Unable to find the requested .Net Framework Data Provider. It may not be installed."

A duplicate entry in machine.config caused the .NET framework to be unable to identify the correct driver to use with Ctree.VisualStudio.Data. Although FairComConfig has logic to find duplicate entries in machine.config, a change in the display name of the driver between V11 and V12 meant it was unable to locate V11 installations.

FairComConfig has been modified to use a different attribute to locate earlier installations: The invariant driver name (ctree.data.sqlclient) is not expected to change among versions.

Upgrading from Previous Editions

The topics in this section will help you upgrade from previous versions of the FairCom Database Engine.

For more information, visit our update guide.

Steps to Upgrade FairCom Server

While FairCom always attempts to maintain backward compatibility whenever possible, transaction logs from earlier versions are generally not always compatible with newer FairCom Server formats.

Removing Transaction Logs and Upgrading the FairCom Database Engine

The FairCom process makes it easy to upgrade the FairCom server using your existing files, as long as you remove prior transaction logs in a safe manner. The steps shown below are appropriate any time you are upgrading. Notice that you will shut down your server two times during this process (steps 2 and 5) to allow all files to be brought to a consistent state.

1. Have all clients cleanly exit from your existing FairCom server.
2. Perform a normal controlled shutdown of the server using one of the methods described here, depending upon your installation:
   • Server Console Window - From the FairCom Server console window click “Control” and then click “Shutdown the Server”
   • Windows Toolbar - Right-click the c-tree Server icon in the Windows Tooltray and choose “ShutDown the Server”
   • Windows Service - From the Windows Control Panel, choose “Administrative Tools”, then choose “Services”. Locate the FairCom Server in the list of services running on your machine. Right-click the c-tree Service and choose “Stop”.
   • Use the client command line utility, ctadmn, and follow the prompts.
   • Use the client command line utility, ctstop.

3. Block the ability of any clients to attach to the FairCom server.
4. Restart the existing FairCom server with no clients attached and allow a successful automatic recovery to take place. This ensures all files are brought to a consistent state in the event there is any data remaining in the transaction logs.
5. Perform another normal controlled shutdown of the FairCom server as described earlier.
6. Remove all existing transaction logs and associated files (L*.FCS, S*.FCS, D*.FCS, I*.FCS, and *.FCT).

Several other types of log dependencies should be considered before deleting transaction logs:

1. Replicas
2. Deferred indexes
3. Record update callbacks

If these options are in use, ensure they have processed all the log data before the logs are deleted. Otherwise the dependent data could be out-of-sync. (2 & 3 don't always use the transaction logs)

7. Copy your new FairCom server directory in its entirety into the existing Server directory. Note you might want to protect your existing ctsrvr.cfg and ctsrvr.set files so you don't loose any custom settings. Also review the new ctsrvr.cfg file accompanying any FairCom server upgrade to leverage new best practice settings.

8. Unblock the ability of any clients to attach.
9. Start the FairCom server in your usual manner and begin using your existing data.

FairCom has added logic to notify you when transaction logs may be incompatible. Please review the section "Detection of Transaction Log Incompatibilities" in the FairCom Server Administrator’s Guide for details.

Upgrade Notes for Developers

This information is intended as a quick reference to demonstrate how easy it is to move from previous versions of FairCom DB database technology to the latest FairCom Database Engine. Because the FairCom Database Engine is a flexible tool that has enjoyed great longevity, it is impossible to anticipate every situation that will be encountered when upgrading. As always, our outstanding engineering team is ready to assist at every opportunity. Feel free to contact your nearest FairCom office for advice and guidance before migrating your application. We have helped successfully migrate many original FairCom DB applications to our very latest versions, sometimes within minutes.

The next sub-sections provide notes about upgrading from previous releases.

Upgrade Notes for Administrators

This section highlights a few notes about upgrading from previous versions of FairCom DB technology to the latest FairCom Database Engine.

GUI Tools

FairCom Database Engine includes a robust set of tools that feature a graphical user interface (GUI). These tools are useful to administrators as well as developers.

Some versions of FairCom Database Engine (e.g., V11) include updates to the SQL engine. This may pose compatibility issues if you attempt to use the new tools with an older FairCom server. We do not recommend connecting new tools to older servers.

Transaction Logs

When upgrading it is advisable to start with new versions of the transaction logs because log formats can change between releases. See Steps to Upgrade the FairCom Database Engine for information about removing the old logs so you can start fresh with your new version.

FAIRCOM.FCS

With V12, FairCom has strengthened it's license file checks. Therefore, before rolling out a V12 production license (ctsrvr*.lic file), we recommend thorough testing on your production machine to ensure your license files are properly sized for your production hardware. Details on CPU counting and licensing can be found here.

FAIRCOM.FCS V9 and older must be recreated:

FairCom DB V12 and FairCom RTG V3 have deprecated an older algorithm that prevents usage of FAIRCOM.FCS files from FairCom DB V9 and older Server lines (and any customers using the prev10logon switch within their V10 and newer ctsrvr*.lic files). The solution is to recreate the FAIRCOM.FCS file by not moving this file forward.

See Adjusting PAGE_SIZE.

Adjusting PAGE_SIZE

Adjusting page size is a complex topic. To learn more details about it, see Important - PAGE_SIZE Conversions.

Notice that a file created with a larger PAGE_SIZE cannot be opened by a system with a smaller PAGE_SIZE. 

Follow these procedures to rebuild the affected files:

1. Shut down the FairCom Database Engine, start it again, then shut it down again to be sure all logs are flushed and a clean final checkpoint has been written.
2. Make a good backup of your databases and your server folder.

3. Edit ctsrvr.cfg and add the PAGE_SIZE keyword if it is not already there:
   PAGE_SIZE 32768
   1. The size should be a power of 2. The maximum page size is 64k (65536).
   2. The current default in V12 and later is 32768.
4. Go to the FairCom data directory.
5. Delete all log and start files (i.e., all files that start with "L00…", "S00…", "D00…"). With a clean shutdown you should not have any "I00…" files in the directory.
6. For V12.0.2 and greater, FairCom DB attempts to convert its internal control files to the larger PAGE_SIZE specified in ctsrvr.cfg, if needed. See FairCom Internal File Conversion below for important details.
   For V12.0.1 and earlier, rebuild the three superfiles used by the FairCom Database Engine using the ctscmp utility including the -sect parameter:
   • Remember that the sect size is the page size/128. For example, if your page size is 32768, use a sect size of 256.
     ctscmp.exe <server directory>\data\ctdbdict.fsd 256
ctscmp.exe <server directory>\data\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd 256
ctscmp.exe <server directory>\data\FAIRCOM.FCS 256
7. Delete the current index files for your data files.
8. Change the indexes for your data files using the ctcmpcif or ctcmpcif.standalone utility including the -sect parameter (which is -256 in the example below):

./ctcmpcif (server directory)/data/ctreeSQL.dbs/MyData.dat ADMIN ADMIN FAIRCOMS -256
./ctcmpcif.standalone (server directory)/data/ctreeSQL.dbs/MyData.dat  -256

9. Fixed-length files: Rebuild the indexes for your fixed-length data files using the ctrbldif utility including the -sect parameter (which is -256 in the example below):

./ctrbldif (server directory)/data/ctreeSQL.dbs/MyData.dat -256

Variable-length files: Rebuild the indexes using ctcmpcif.exe as follows:

ctcmpcif.exe <your application data directory>\your_file.dat 256

1. Restart the server and verify no errors were logged in <faircom>/data/CTSTATUS.FCS.
2. Connect to your data files and verify no errors were returned.

FairCom Internal File Conversion

When the PAGE_SIZE server configuration option is changed, FairCom V12.0.2 and later attempts to apply this change to all FairCom server-controlled files. At server startup we check the page size of the following FairCom files:

• ctdbdict.fsd
• Database dictionaries
• FAIRCOM.FCS
• SEQUENCEDT.FCS
• DFRKSTATEDT.FCS
• SYSLOGDT.FCS
• REPLFFCHGDT.FCS
• REPLOGDT.FCS
• REPLOGSHIPDT.FCS
• REPLSTATEDT.FCS
• REPLSYNCDT1.FCS
• REPLSYNCDT2.FCS

If any do not match the newly specified PAGE_SIZE, a backup copy of the existing file is made with an .FCB extension. The server process then attempts to invoke the FairCom ctscmp.exe (ctscmp on Linux/Unix) superfile rebuild utility with the new configured PAGE_SIZE.

Superfiles require that the PAGE_SIZE at open matches the PAGE_SIZE at file creation time, or a KSIZ_ERR (40) or SPAG_ERR (417) error occurs at file open. For FAIRCOM.FCS, this prevents the server from starting when operating with a different configured PAGE_SIZE setting.

A new configuration keyword has been added to control the automatic page size conversion at startup:

PAGE_SIZE_CONVERSION {YES|NO}

Default: YES

Limitations:

1. We expect ctscmp.exe (ctscmp on Linux/Unix) to exist in the process working directory and have permission to be executable by the server process. If this utility does not exist, the server fails to start with the following messages likely logged to CTSTATUS.FCS:

 - User# 00001	Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768
 - User# 00001	Did not find ctscmp in working directory for Superfile conversion: 2
 - User# 00001	Could not process User/Group Information in FAIRCOM.FCS: 417
 - User# 00001	Could not initialize server. Error: 417
 - User# 00001	O1 M0 L74 F417 P0x (recur #1) (uerr_cod=417

2. The LOCAL_DIRECTORY keyword must be set and must be different from the working directory. Otherwise, ctscmp.exe will encounter a TCOL_ERR (537) and fail.
3. Superfile conversion occurs only after auto-recovery. Recovery is likely to fail if run with a different PAGE_SIZE setting. We expect any needed recovery should occur before making a major configuration impacting the physical attributes of critical operational files such as authentication information and SQL database dictionaries (system tables).
4. NO FURTHER ATTEMPT IS MADE to convert any other existing indexes. All other application created indexes will require manual rebuilding to convert to a new page size. See the steps below for the best practice procedures. The following messages may be found logged to CTSTATUS.FCS after a successful conversion:

 - User# 00001	Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768
 - User# 00001	Backup created ../data/\FAIRCOM.FCS => ../data/\FAIRCOM.FCS.1621883226.FCB
 - User# 00001	Superfile conversion successful
 - User# 00001	Wrong PAGESIZE for ctdbdict.fsd, attempting to convert file from PAGESIZE 8192 to 32768
 - User# 00001	Backup created ../data/\ctdbdict.fsd => ../data/\ctdbdict.fsd.1621883232.FCB
 - User# 00001	Superfile conversion successful
 - User# 00001	Wrong PAGESIZE for .\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd, attempting to convert file from PAGESIZE 8192 to 32768
 - User# 00001	Backup created ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd => ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd.1621883237.FCB
 - User# 00001	Superfile conversion successful

Once correct server operations are confirmed after conversion, the *.FCB files can be removed and deleted.

More about Upgrading

The same core database technology found in prior releases powers FairCom Database Engine. With an emphasis on ease of use and unfettered development, the FairCom Database Engine is the same core engine you've come to depend and rely upon for many years in your existing and successful applications.

There are many ways to move forward with FairCom technology from existing applications and previous versions of FairCom DB. See which pathway best fits your situation and see how easy it is to begin using FairCom DB. As FairCom database technology is extremely flexible and has been deployed in a huge number of diverse applications, there may be subtle issues unique to any upgrade.

Please don't hesitate to contact your nearest FairCom office for assistance. Our experienced engineering team is always willing to assist in any way possible.

Are you an existing FairCom DB developer?

• Link your existing application to the FairCom multithreaded library, mtclient.lib, included in the /lib directory of your installation.
• Copy or move your existing data and index files to the working directory of the FairCom server, which in V12 has changed from /bin/ace/isam or /bin/ace/sql to <faircom>/server. You could also consider the LOCAL_DIRECTORY configuration keyword to point to your existing data location. Be aware that you will need to remove existing transaction log files as previous versions of these files are incompatible with newer versions. Follow the steps in this section to ensure a clean start with FairCom DB.
• Continue using your existing application!

Are you an existing c-tree Plus Standalone developer?

FairCom DB offers a client-server only version. In many cases, you can easily make your existing data work with the FairCom server and gain all of the benefits of FairCom technology. Dynamic backups, advanced caching, and SNAPSHOT statistics monitoring are just a few of the great features available with the FairCom server engine. To begin using this technology in your application follow these simple steps:

• Link your existing application to the multithreaded library, mtclient.lib.
• Copy or move your existing data and index files to the working directory of FairCom DB. You could also consider the LOCAL_DIRECTORY configuration keyword to point to your existing data location. Be aware that you will need to remove existing transaction log files as previous versions of these files are incompatible with FairCom DB. Follow the steps in this section to ensure a clean start.
• Rebuild indexes to take advantage of larger page sizes. Standalone models default to a 2048 byte index node size, while FairCom Servers since V8.14 default to 8192 bytes. Increasing this allows more keys per node, as well as larger keys, that frequently arise when creating complex indexes via SQL.
• Switch your initialization to InitISAMXtd() and add the client user name and password to access the server. Use the Xtd() Function API calls that allow the user credentials and server name to be specified. You can include these in your standalone builds, as they are simply ignored in those models.
• Consider multithreading your application for enhanced scalability and performance.
• Access the power of the FairCom DB in your application immediately.

Do you wish to use existing FairCom data with FairCom SQL?

• Ensure your data is compatible with FairCom DB SQL. FairCom DB SQL requires proper DODA and IFIL structures to be present. Use the FairCom DB Information utility, ctinfo, included with the product, to check for the presence of these required resources.
• Use the SQL Import utility, ctsqlimp, to import your existing data and indexes to FairCom DB SQL. The following topic, Let Your Existing ISAM Applications Co-Exist with SQL, provides the steps needed for this data access addition.
• Begin using any FairCom DB SQL interface technology with your existing FairCom DB data!

Let Existing ISAM Applications Co-Exist with SQL

FairCom SQL has been designed from its core to provide as much access as possible to all existing FairCom data. For most applications, it is as simple as linking the data to the FairCom Server system tables using the Table Import Utility, ctsqlimp. Not only does this give you the ability to view and modify your tables with SQL, you also retain the ability to continue using your existing application!

• Advantages: No changes are made to the data and index file definitions, so the existing c-tree application can access the data without changes to the application.
• Considerations: Some higher-level SQL capabilities that require special internal fields, indexes, and file modes will not be supported unless the files and applications are adjusted to provide these requirements.

Example ISAM Application with Proper SQL Constructs

Table Definition Requirements

To take advantage of the ability to co-exist with SQL, certain requirements must be met to ensure compatibility.

• Tables must contain IFIL and DODA structures. These can be added after the fact for existing files and are inserted automatically for files created by c-treeDB and FairCom SQL.
• An ISAM application must use corresponding c-tree Plus data types (as defined in the DODA) as in the c-tree Plus - SQL data type mapping. For example a CT_CHAR field type is used in FairCom SQL to store a 1-byte integer.

CT_ARRAY fields are imported by default as a FairCom SQL Binary field. FairCom SQL expects the first four bytes of a binary field to specify the length of the field. When you create a table with FairCom SQL these four bytes are automatically created and maintained for you. When considering a CT_ARRAY field from c- tree Plus, you must explicitly include these four prefix bytes and assign the appropriate value. An example of how to properly handle this field is demonstrated in the example code at the end of this article. Should you have existing incompatible c-tree Plus CT_ARRAY fields to import into FairCom SQL, please contact your nearest FairCom office for suggestions and advice. We're here to help you.

• The table must have either TRNLOG or PREIMG in its file mode to use the ROLLBACK WORK and integrity constraint capabilities.
• Superfiles are supported by FairCom SQL; however, it's not possible to create a superfile using SQL. Please create using another API like the C Database API.
• In order to properly handle NULL, the table must contain the $NULFLD$ field, a hidden field generated by c-treeDB at creation time. Tables created with the c-treeDB interface (used with c-treeSQL) have a hidden field, $NULFLD$, which is used to determine if each user-created field in the record buffer has a NULL value. c-treeSQL requires this capability to implement constraints. c-treeDB and FairCom SQL will access tables without the $NULFLD$ field, but the table's fields will always return a non-NULL status.
• In order to properly handle JOINS referencing ROWID, the table should contain the $ROWID$ field (a hidden field generated by the c-treeDB at creation time). c- treeDB and FairCom SQL should work with tables without the $ROWID$ field, and will use the record offset as the ROWID tuple identifier. SQL statements like select * from table where rowid > '4' will fail because using record offset as ROWID will give us record offsets instead of sequential numbers.

Adding a DODA to an Existing Data File

To use the FairCom SQL Server with existing ISAM files that do not already have a DODA resource, add a DODA to each file. This is done most easily with a developer-created utility that opens each file and calls PutDODA to insert the required resource into that file. The utility should accomplish the following tasks:

• Include a data object definition array (DODA) which is simply an array of DATOBJ structures, as defined below.
• Open each data file in ctEXCLUSIVE mode.
• Call PutDODA() for each file to insert the corresponding DODA resource.
• Close the files.

A DODA is a data object definition array. Each element of the array is comprised of a structure of type DATOBJ. Only three of the first four fields of the DATOBJ are required for standard c-tree Plus data files. DATOBJ is defined as follows:

typedef struct {
 pTEXT fsymb; /* ptr to symbol name */
 pTEXT fadr; /* adr of field in record buffer */
 UCOUNT ftype; /* type indicator */
 UCOUNT flen; /* field length */
 ...
} DATOBJ;

• fsymb points to a unique symbolic name for the field and should not be NULL.
• fadr is not used by c-tree Plus (its value is ignored).
• ftype is one of the field types specified in the "Field Types" table.
• flen is set to the field's length for fixed length fields, or the known maximum for varying length fields with a known maximum length, or zero for varying length fields without a known maximum length. If the field type has an intrinsic length, which is true for types CT_CHAR through CT_DFLOAT, a zero length is automatically replaced by the intrinsic length.

Given a data record with the structure:

struct {
 TEXT zipcode[10]; /* Zip code */
 LONG ssn; /* social security # */
 TEXT name[50]; /* name */
} DATA_FORMAT;

The corresponding DODA would be defined as:

DATOBJ doda[] = {
 {"ZipCode",NULL,CT_FSTRING,10},
 {"SocialSecurity",NULL,CT_INT4},
 {"Name",NULL,CT_STRING,50}
};

The PutDODA() call inserts the DODA object as a resource into the data file. The function is declared as follows:

PutDODA (COUNT PUTDODA(COUNT datno,pDATOBJ doda,UCOUNT numfld))

PutDODA() assigns the contents of a data object definition array (DODA) to data file datno , which must be opened in ctEXCLUSIVE mode. doda points to the beginning of the DODA as described above. The numfld parameter indicates how many data fields are in the DODA, three in the example above. See review the PutDODA() function description and "Record Schemes" in the FairCom Programmer’s Reference Guide for additional details. Call FairCom for assistance if needed.

Index Definition Requirements

• If an index contains a segment consisting of a "partial field" (i.e., does not use the c-tree Plus Schema segment modes or the segment starting offset and the segment length are different from the field starting offset and the field length) c-treeSQL cannot access this index, even though the index is still properly updated by c-tree. You will need to create a new index in c-treeSQL composed of the corresponding columns.
• If there is more than one logical index in one physical index file, the DROP INDEX and the DROP TABLE commands will not work properly.

ALTER TABLE may not work correctly if tables contain index segments that do not start at field boundaries and/or span over several fields. 

For example, if a field is deleted from the table, and this field is part of an index segment that spans over several fields, c- treeSQL may not know how to adjust the index segment length after the field is deleted from the table. The resulting index definition may not be correct. Tables with unusual characteristics may also not work correctly and the altered table may inherit characteristics preventing them from working in the original application.

Example

The following application defines a typical c-tree Plus ISAM data file and index with the proper IFIL and DODA resources necessary for use with c-treeSQL. In addition, it demonstrates the proper construction of a CT_ARRAY field to be imported into a c-treeSQL database table as a BINARY field.

See Example ISAM Application with Proper c-treeSQL Constructs.

After executing this application, run the utility to link the table to c-treeSQL:

#ctsqlimp isam_table.dat -u ADMIN -a ADMIN

Finally, run the utility to issue c- treeSQL statements against the table:

#ISQL -u ADMIN -a ADMIN ctreeSQL
ISQ>SELECT * FROM isam_table;

Troubleshooting ISAM to c-treeSQL Problems

The easiest way to avoid common problems when importing c-tree Plus data files into c-treeSQL is to copy these files into the c-treeSQL database directory, typically located in the c-treeSQL Server directory. This gives the server direct access to the files. However, it is possible to link any c-tree Plus data file from any location. c-treeSQL Server access to the file is completely relative. The most common problem encountered is an FOPN_ERR error (12) from ctsqlimp when importing the table. In most cases, this is simply the c- treeSQL Server's inability to resolve the file's relative location from the c-treeSQL dictionary files. The most straightforward way to address this issue is to specify the full pathname when specifying the data file to import.

For example, to import a c-tree Plus data file existing in another directory different from the c-treeSQL Server, execute the ctsqlimp command as follows:

#ctsqlimp c:\old_data\datafile.dat -d c-treeSQL -s FAIRCOMS -u ADMIN -a ADMIN

This will link the existing data file in place to the c-treeSQL Server. You can now query, add, and update the data with standard c-treeSQL statements.

Important!

While you can import data from another c-tree Server location into the c-treeSQL Server, keep in mind you can only use ONE of the servers to access the data. It is not possible to access a c-tree data file from multiple c-tree Servers simultaneously. With the c-treeSQL Server this is not a problem; you can continue to use your existing ISAM application with the new c-treeSQL Server!

Upgrading V6 Applications

As always, the FairCom DB is designed for immediate use right out of the box.

Applications written for c-tree Plus V6 that include ctreep.h as required will compile and link properly right out of the box. Your application will function exactly as before using the latest FairCom DB library. ctreep.h automatically includes the ctv6v7.h header (described in “#defines”).

Developers wanting to use HUGE, segmented files or other functionality made available with the Xtd8 file format (see File Formats in the FairCom Programmer's Reference Guide) first introduced in V7 must also:

• Use the 8-byte file creation functions to create new extended files.
• Change IFIL definitions for indexes allowing duplicates to store 8-byte offsets for indexes referencing HUGE files.
• Convert existing data files using the ctcv67 utility.

Note that the ctoptn.h file is now created in a custom.xxx directory where .xxx represents the operational model. This file was previously created in the custom directory.

Duplicate Keys

For indexes associated with Standard data files, a duplicate key length includes 4 bytes for the associated record position which is used to break ties. If an index is created for a HUGE data file, then the key length must include 8 bytes for the associated record position.

#defines

The FairCom DB #define constants were modified to use a uniform style of ctWXYZ by appending ct to the beginning of all FairCom DB defines. For example, TRNLOG is changed to ctTRNLOG.

Two header files, ctv6v7.h and ctv7v6.h, provide compatibility.

Add the ctv6v7.h compatibility header file to applications written for c-tree Plus V6.x to automatically change the old defines to the new defines to allow the application to be compiled with the latest version of FairCom DB.

Add the ctv7v6.h compatibility header file to applications written for the latest version of FairCom DB to automatically change the new defines to the old defines to allow the application to be compiled with c-tree Plus V6.x. This header provides “rename #defines” to set the symbols back to their V6 convention.

The following #defines were changed:

Converting c-tree V4.1F-V4.3C Applications

The conversion of an existing c-tree V4.3 application program involves two main aspects: file conversion and program conversion. The files from c-tree V4.3 are not compatible with the FairCom DB file format. Programs may be compatible, but many will require some modification.

Application Conversion Details

• data & index files
  Data files are converted using the utility program ctcv43 provided with FairCom DB. Index files associated with these data files should then be rebuilt using the FairCom DB function, RebuildIFile(). Stand-alone index files are converted with the programs ctin43 and ctindx provided with FairCom DB. ctin43 must be linked with your V4.3 library. It produces a flat key file for each index. ctindx is linked with your FairCom DB library. It creates a FairCom DB index from the flat key file produced by ctin43.
• ISAM information
  ISAM Parameter files and Incremental ISAM Structures are compatible. The tfilno field has been added to the IFIL structure. The aidxnam, altseq, and pvbyte fields have been added to the IIDX structure. See Incremental ISAM Structures in the FairCom DB Programmer's Reference Guide for more information on these structures. With parameter files, make sure the RTREE setting in ctoptn.h is correct.
• header info
  FairCom DB applications cannot directly access the index file headers via the(ct_key + filno)->member reference. Instead, use the GetCtFileInfo() and GetSymbolicNames() functions, part of the FairCom DB library, to get the information. For example, the data record length is accessible with the following expression in c-tree V4.3: (ct_key + filno)->reclen. In FairCom DB the record length is returned by the function call GetCtFileInfo(filno,RECLEN). GetCtFileInfo() returns a long integer since it is also used to return header values such as the file size. All references of the form “extern CTFILE *ctnum;” must be removed. If you are using such a statement, then you must rely on the GetCtFileInfo() and GetSymbolicNames() functions instead.
• length parms
  All length related parameters have become 4-byte values. We now provide function prototypes which should catch (and/or correct) this change. For example, in REDVREC(datno,recptr,bufsiz), bufsiz is now a 4-byte quantity.
• Function Returns and Parameter Changes
  The following functions have undergone parameter changes between c-tree V4 and c-tree Plus V6:

• frmkey
  If you use the BuildKey() function, short name frmkey, it now takes an additional parameter specifying the length, as a 4-byte quantity, of the record buffer from which the key is extracted. See the function prototype for BuildKey() in CTDECL.H.
• current ISAM record
  cur_recno[ ] and cur_image[ ] no longer exist. Also, it is no longer necessary to use two buffers for record updates, although using two buffers will NOT cause any problem for FairCom DB. FairCom DB maintains the necessary information internally for each user. To determine the current ISAM record position, use GETCURP(datno) instead of cur_recno[datno]. To reset the current ISAM record after a successful rewrite, use UPDCURI(datno,SWTCURI) instead of manipulating cur_recno and cur_image. See “ResetRecord” or “CurrentFileOffset” in the FairCom DB Programmer's Reference Guide for additional capabilities. To set the current ISAM record to a specified byte position (and/or record image) use SETCURI(datno,recbyt,recptr,bufsiz), where recbyt is required and specifies the record position. If non-null, recptr points to a record buffer containing an image of the data record. If non-zero, bufsiz (a 4-byte value), specifies the length of the record buffer. If recptr is non-null and bufsiz is passed as 0L, then bufsiz is assumed to be the record length defined for the file at the time the file was created. For variable length files, this corresponds to the fixed length portion of the file.
• key segment info
  If you have accessed key segment information (such as a segment length or segment mode), you can now retrieve it via a call to:
  ctgetseginfo(keyno,segment_no,mode)
  where segment_no is a zero-based segment number, and mode is one of the following:
• ctSEGPOS - segment offset
• ctSEGMOD - segment mode
• ctSEGLEN - segment length
• #includes
  Be sure to change the FairCom DB include files to the following:
  #include “ctreep.h”
  Note that ctaerr.h includes the definitions for uerr_cod, isam_err and isam_fil. It is not necessary to include ctifil.h (as specified in the manual) since it will be included automatically.
• old key types
  Key types 1, 2 and 3 have been phased out. They supported integer and floating point keys. You can use appropriate key segments to support all of these types. Change the key type to 0, unless you desire key compression, and modify your ISAM key segment modes. If your application only uses the low level functions, you can transform the keys as required using the TransformSegment() function described in the FairCom DB Function Reference Guide.
• Lock Table Size
  c-tree V4.X utilized an internal lock table that supported 32 concurrent locks by default, #define MAX_LOCKS 32. To alleviate this artificial lock limit, FairCom DB dynamically allocates a lock list instead of a more static lock table. The number of locks available to FairCom DB is now constrained by available system memory.
• TRANSACTION
  The TRANSACTION() function supported in c-tree V4.3, for purposes of keeping a Server from shutting down in the middle of an application procedure, now simply returns without error. It causes no action to be taken. FairCom DB supports extensive transaction processing in the Server mode via the Begin(), Abort(), Commit(), SetSavePoint(), and RestoreSavePoint() functions.
• TFRMKEY
  FairCom DB supports the automatic calling of TransformKey(), short name TFRMKEY(), to transform target key values used in ISAM level search routines. If you use the traditional FairCom DB functions to initialize FairCom DB, InitCTree(), InitISAM(), OpenISAM(), or CreateISAM(), then this automatic feature is disabled and your existing application will be unaffected. If you use the extended forms of these functions (InitCTreeXtd(), InitISAMXtd(), OpenISAMXtd(), or CreateISAMXtd()), then you must set the userprof parameter to USERPRF_NTKEY to disable the automatic calls to TransformKey(). Otherwise, your existing calls will be followed by the automatic calls, which will cause problems.
• FPUTONLY
  There is no longer a FPUTONLY disk I/O protocol. FairCom DB now lets you select on a file-by-file basis whether or not to force updates directly to disk. OR the ctWRITETHRU constant (64) into the file mode to get the same affect as the FPUTONLY I/O protocol.
• DOSFLUSH
  DOSFLUSH support is being phased-out. As distributed, DOSFLUSH will only become effective in the FPUTFGET, Standalone Multi-user, configuration. If necessary, add DOSFLUSH to your CTOPTN.H configuration file. CTOPT2.H contains preprocessor commands that turn DOSFLUSH off if NOTFORCE is selected.) Transaction processing provides a much more effective means to ensure the consistency and completeness of the data.
• r-tree^® / d-tree^™
  Applications using Version 1.1 of the r-tree Report Generator or Version 3.1 of the d-tree Development ToolBox require updated versions of these development tools.

ctin43 - Create flat key file from V4.3 index file

Usage

ctin43 <index file> <index member #> <flat key file>

Description

ctin43 must be compiled and linked with a c-tree V4.3 application library. It will not work with a FairCom DB application library.

ctin43 runs in single user mode, and dumps the keys in order, for the designated index member, to a flat file. It uses the same form as ctdidx and ctindx. Convert an index from V4.1F-V4.3C to FairCom DB by first using ctin43 and then ctindx.

ctitop - Create OTP and Parameter Files

The ctitop tool creates an OTP file and a parameter file from IFIL and DODA resources read from a c-tree data file.

Usage

ctitop [-<sectors>] <filename>

ctpotp - Parameter to OTP conversion

Usage

ctpotp <parameter file name> [<user profile bitmask>]

Description

This utility takes a parameter file and produces the corresponding OTP file(s) used by the FairCom Drivers. If no record schema (DODA) is present, a dummy schema is created.

ctptoi - Parameter to IFIL conversion

Usage

ctptoi <parameter file name> [<user profile bitmask>]

Description

This utility opens the parameter file whose name is specified as a command-line argument and writes an IFIL (based upon the parameter file definitions) to each data file referenced in the parameter file. ctptoi is designed for use as a standalone non-server utility.

ctpkey - Print key utility

Usage

ctpkey <parameter file name> [<# of entries>]

Description

ctpkey prints out 1 or more key value(s), up to the optional # of entries, from each index file in the parameter file, showing how each key segment is defined. The keys are displayed in hex and ASCII. This utility will run in all modes of operation. It is useful if you are having trouble retrieving data. ctpkey will allow you to see how a key value is constructed from the data.

By changing a #define, ctpkey.c can be changed from a main line program to the callable function ctpkey(), with the same parameters as OpenIFileXtd().