Monitoring FairCom DB

The FairCom Server tracks and reports a wealth of performance-oriented statistics. The server's performance snapshot capability enables the capture of performance monitoring data using a combination of configuration file options and the SnapShot FairCom DB API function. The performance data can be captured automatically at specified intervals or on demand in the following ways:

• The ctstat utility provides a command-line interface to the SnapShot API function, supporting output of statistics at the FairCom Server system, user, file, and function levels.
• Configuration file options support automatic performance snapshots by specifying the interval between automatic snapshots and the contents of the snapshots. The performance data captured by automatic snapshots is written to the FairCom Server system event log (SYSLOG) files.
• Use DIAGNOSTICS options to capture the automatic system snapshot data to the human-readable SNAPSHOT.FCS file.
• The SnapShot API function can control automatic snapshots, override configuration options (if any), and capture on-demand performance data. Captured data can be sent to:
  • SYSLOG files
  • the human-readable SNAPSHOT.FCS file
  • the calling program

The following sections discuss how to use the performance monitoring abilities of the FairCom Server.

Performance Monitoring Using the ctstat Utility

The ctstat utility is a client utility used to display statistics collected by the FairCom Server. It is found in the client folder of the server installation and demonstrates the use of the SnapShot function.

The section titled Statistics Monitoring Utility - ctstat (ctstat - Statistics Utility, ctstat Statistics Utility) describes the reports that this utility produces and how to use the utility to generate these reports.

Performance Monitoring Using Server Keywords

This section describes automatic performance snapshot logging using FairCom Server configuration keywords.

Automatically Logging Performance Snapshots

The FairCom Server supports automatically logging performance snapshots to the SYSLOG files and to the human-readable file SNAPSHOT.FCS. The following sections describe how to enable automatic snapshots.

Automatic Logging to the Server System Event Log

The SNAPSHOT_INTERVAL keyword enables automatic snapshots at specified intervals:

SNAPSHOT_INTERVAL  <minutes>

By default, only the system snapshot is captured. To add user or file-specific snapshots to the data captured, use one or more of the following configuration entries:

SNAPSHOT_USERID     <user ID>

SNAPSHOT_FILENAME   <file name>

Files and users added to the snapshots are said to be activated. Users and files may be activated whether or not the automatic snapshots are turned on in the configuration file. However, the activation has no effect until snapshots are written to the SYSLOG files.

The <user ID> and <file name> arguments may include wildcard matching characters: “*” matches an arbitrary number of any characters, and “?” matches exactly one of any character. A pattern of simply “*” matches any user or file name. For example, the following keywords activate all users, any file ending in “.dat”, and the file journal.idx:

SNAPSHOT_USERID     *

SNAPSHOT_FILENAME   *.dat

SNAPSHOT_FILENAME   journal.idx

User IDs are not case sensitive. File name case sensitivity depends on the platform. For example, Windows is case insensitive and Unix is case sensitive. The file names activated must match the file name used to first open the file. In particular, paths used in the activation list and during the call to open the file must match.

Automatic Logging to SNAPSHOT.FCS

Write system snapshots to the human-readable SNAPSHOT.FCS text file with the following DIAGNOSTICS options:

DIAGNOSTICS   SNAPSHOT_SHUTDOWN

DIAGNOSTICS   SNAPSHOT_AUTOMATIC

DIAGNOSTICS SNAPSHOT_SHUTDOWN writes a system snapshot to SNAPSHOT.FCS at the start of the server shutdown process. DIAGNOSTICS SNAPSHOT_AUTOMATIC writes any automatic snapshots to SNAPSHOT.FCS instead of to the SYSLOG files. However, only the system snapshot is written. Snapshots for activated users and/or files are ignored.

Performance Monitoring Using the SnapShot API

FairCom Server configuration options provide an easy way to enable automatic performance snapshots. Additional flexibility and access to the complete set of statistics maintained by the FairCom Server is available programmatically through the SnapShot FairCom DB API function. This API function:

• Controls automatic snapshots, overriding configuration options (if any).
• Writes snapshots to the SYSLOG files on demand, whether or not automatic snapshots are active.
• Writes system snapshots in human-readable form to SNAPSHOT.FCS.
• Returns snapshot data on-demand to the calling application program.

See Also

• A subset of the SnapShot statistics is available using server keywords. See Performance Monitoring Using Server Keywords.
• For information about using the SnapShot function, see Performance Monitoring Using the SnapShot API and SnapShot in the FairCom DB Programmer's Reference.

FairCom Server Status Monitoring Utility, ctsysm

The FairCom Server status monitoring utility, ctsysm, facilitates the monitoring of error, warning, and informational messages logged to the server status log, CTSTATUS.FCS, by FairCom DB. Using this utility, the FairCom Server status log can be monitored by an automated external process.

Using a customizable configuration file, the ctsysm utility provides:

• A well-defined format for server messages to make them readable by automated systems
• Additional context as to whether the message is an error or purely an informational message
• The FairCom Server subsystem that is involved
• A recommended action

The following diagram shows a conceptual view of the operation of the ctsysm utility:

Using the ctsysm Utility

To use the ctsysm utility, the FairCom Server must be configured to log messages to the system log. This can be achieved by adding the following keyword to the server's configuration file:

SYSLOG CTSTATUS

With this keyword in place, the server logs each entry in the CTSTATUS.FCS file to the system log files (SYSLOGDT.FCS and SYSLOGIX.FCS).

The utility can read the system log from the beginning each time it starts up, or it can save its current position and start again from that set position by specifying the -f command line option.

The following is the supported command line usage of the FairCom Server Status Monitoring Utility:

ctsysm [-s svn] [-u uid] [-p upw] [-r rpt] [-c cfg] [-f fil] [-l log] [-e]

• -s svn - FairCom Server name
• -u uid - user name
• -p upw - user password
• -r rpt - repeat interval in seconds (for example, -r 1 is a one-second delay before checking for new messages)
• -c cfg - the name of the utility’s configuration file (default is ctsysm.cfg)
• -f fil - save/restore state to file (see notes below)
• -l log - status log name (SYSLOGDT.FCS) (see notes below)
• -e - direct ctsysm error messages to standard output in the same format as messages read from the server status log:

<timestamp> E0000 SYSM MSGT ctsysm ctsysm <error message text>

Notes:

-f causes the utility to save the current state of the utility to the specified file when the utility shuts down and to read the current state of the utility from the specified file when starting up. This option can be used to cause the utility to start reading after the last message it read from the system log instead of starting at the beginning of the system log.

-l specifies the name of the file to read, either SYSLOGDT.FCS (for monitoring server status using the server’s system log) or CTSTATUS.FCS (for monitoring server status using CTSTATUS.FCS directly). Using SYSLOGDT.FCS has the advantage that it supports the -f option and can be used from a remote system, while using CTSTATUS.FCS has the advantage that all status log messages are captured (note that the server may not log some startup and shutdown messages to the system log because the system log is not active during part of startup and shutdown; perhaps the most critical time of interest that the system log is not active is during automatic recovery).

Authentication File

This utility supports the use of an encoded password file. Encoded password files keep user IDs and passwords from plain view when the utility is used within a script file. They are created with the ctcmdset utility. The plain text form of the file should be:

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

Example

The following command causes ctsysm to monitor the FairCom Server system log for the server FAIRCOMS (-s FAIRCOMS) for status messages, checking for new messages every second (-r 1), saving the position of the last entry read to the file ctsysm.log (-f ctsysm.log).

ctsysm -s FAIRCOMS -u ADMIN -p ADMIN -r 1 -c ctsysm.cfg -f ctsysm.log

The ctsysm utility outputs messages in the following format:

<timestamp> <code> <subsystem> <action> <text>

Sample output from the utility is shown below (each message is output as a single line but is shown split into two lines here):

Thu Jul 29 16:44:04 2004 I0455 STUP NONE                    - User# 01

   Alternative server name...  FAIRCOMS

Thu Jul 29 16:44:04 2004 I0492 STUP NONE                    - User# 01

   Compatibility bit maps: 00000000 00002000x

Thu Jul 29 16:44:04 2004 I0491 STUP NONE                    - User# 01

   Diagnostic bit maps: 00000000 00000000x

Thu Jul 29 16:44:04 2004 I0490 STUP NONE                    - User# 01

   64-bit File Address Support

Thu Jul 29 16:44:04 2004 I0489 STUP NONE                    - User# 01

   6 Byte Transaction Numbers

Thu Jul 29 16:44:04 2004 I0488 STUP NONE                    - User# 01

   NOWAIT usrsema enabled

Note: When the repeat (-r) option is used, the utility can be stopped by sending it a SIGINT signal.

ctsysm Configuration File

The configuration file, ctsysm.cfg, contains server messages classified by the server subsystems that generate the log entries, the recommended actions, and details for each message. The configuration file is simply a text file, an abbreviated copy of which is shown later in this document. The configuration file can be used as-is, although some of the default recommended actions are general and administrators may want to customize these.

The subsystem list consists of entries in this format:

<subsystem_keyword> <subsystem_description>

The action list consists of entries in this format:

<action_keyword> <action_description>

The messages are in the following format:

<code> <subsystem> <action> <num_lines> <text>

where <code> is a 5-character code beginning with a message type and followed by a 4-digit message number (for example, E0001). Possible message codes include:

• F: Fatal error
• E: Error
• W: Warning
• I: Information
• U: Unclassified

Because the configuration file is a text file separate from the server executable, it can easily be updated as new or application-specific information about specific messages becomes available, without having to update the server executable.

ctsysm Configuration File Sample

Below is a portion of a sample of the current configuration file
(<install path>\source\ctsysm.cfg):

[Version]

FairCom Server Status Log Monitor Configuration File V12

[Subsystems]

CADM Connection administration

COMM Communication

DIAG Diagnostic information

DDMP Dynamic dump

FMNT File maintenance

.

.

.

[Actions]

CCFG The server found an unsupported or invalid option in the server configuration file. Check the server configuration file.

CFGW An option specified in the server configuration file might lead to undesired behavior. Review the message to understand the situation.

CKDS The operation failed due to insufficient disk space. Free up disk space to allow the operation to complete.

.

.

.

[Messages]

;==================================================

;Server configuration messages

;==================================================

F0001 SCFG CCFG 1 Log set range too large

F0002 SCFG CCFG 1 Cannot read existing encrypted logs.  Enable log encryption to proceed.

F0003 SCFG CCFG 1 Bad COMPATIBILITY argument

.

.

.

;==================================================

;Server startup messages

;==================================================

F0041 STUP SUPT 1 D0000000.FCS:

F0042 STUP SUPT 1 I0000001.FCS

F0043 STUP SUPT 1 System monitor queue creation error

.

.

.

;==================================================

;Diagnostic information messages

;==================================================

F0135 DIAG SUPT 1 Unexpected internal c-tree(R) error #

E0136 DIAG SUPT 1 relaim_queue processing error

E0137 DIAG SUPT 1 SnapShot event error

.

.

.
 

Server System Event Log Keywords

FairCom DB optionally maintains a system event log, SYSLOG. This is maintained in two system files: SYSLOGDT.FCS and SYSLOGIX.FCS. These files comprise a FairCom DB data file and index pair with a record for each recordable system event. Unlike the text based CTSTATUS.FCS, SYSLOG can be encrypted such that entries cannot be added, deleted, or modified with a simple text editor, and vendors can log application specific entries.

The System Event Log contents are controlled by SYSLOG configuration keywords in ctsrvr.cfg, the ctsrvr.set settings file, or from the command line. They are entered as pairs in the form of: SYSLOG <keyword>. As many of these pairs as desired may be used at the discretion of your application vendor.

Current SYSLOG options include:

ctsmon - Server Heartbeat Monitor

ctsmon is used to check the current running status of a FairCom DB Server. Frequently, this is the "heartbeat" monitoring component in a clustering failover solution.

Usage

ctsmon [-u name] [-p pw] [-s s1,s2,...] [-t nn] [-c nn] [-d nn] [-o api_option]

where

• -u name - User name to connect to the server with. Default is none.
• -p pw - User password. Default is none.

It is assumed that all servers in the server list (-s) have the same user ID and password.

• -s Server_name - Name of the Faircom server to log into.

The server name has the form of name[@host]. If no @host is given the local host is assumed. @host can also be given as using an I/P address Example: FAIRCOMS@192.168.1.1 Default is FAIRCOMS

• -t nn - Time out value in seconds. Default is 30 seconds.
• -c nn - Loop count for checking the server. This parameter has three possible values:
  • +nn - Exit after nn times regardless if the server is up or down.
  • 00 - Loop forever regardless if the server is up or down.
  • -nn - Loop forever while the server is up, if unable to log into the server, exit after nn loops.
• -d nn - Set debugging trace level to n (0 to 5).
• -o api_option - An optional API call to retrieve additional information from server demonstrating the server is functional.

api_option is one of the following:

• appname - First APP_NAME_LIST entry
• curtime - Current server time in seconds since 1970
• commits - Current number of database commits (ctGSMS.sct_trend)

Return status

-1 - Usage error, or internal error.

0 - Server responded.

+n - Error returned from InitISAMXtd call.

Description

This utility is intended for use with cluster management systems such as the Open Cluster Framework (OCF) and Pacemaker on Linux, and Solaris and Microsoft cluster managers. This is usually run in the background as a daemon process. It is a useful utility to use with any monitoring the current availability status of a FairCom DB server process.

The availability check is a basic connection/disconnection attempt. If the connection succeeds, it is assumed the server is active and available. If a further check such as the ability to retrieve data is required, the api_option can be used to trigger that request.

The source code of this utility is provided in the FairCom DB SDK and can be further customized for specific usages.

Example

ctsmon -u ADMIN  -p ADMIN  -s 10.6.1.1  -t 20  -c 3