The following steps will help you detect the source of an application failure.

1. Check any Oracle Tuxedo system warnings and error messages in the user log (ULOG).
2. Select the messages you think most likely reflect the current problem.

   Note:

   The catalog name and the number of each of message, so you can look up the message in System Messages. The manual entry provides:
   • Details about the error condition indicated by the message
   • Recommendations for recovery actions
3. Check any application warnings and error messages in the ULOG.
4. Check any warnings and errors generated by application servers and clients. Such messages are usually sent to the standard output and standard error files (named, by default stdout and stderr, respectively).
   • The stdout and stderr files are located in the directory defined by the APPDIR variable.
   • The stdout and stderr files for your clients and servers may have been renamed. (You can rename the stdout and stderr files by specifying -e and -o in the appropriate client and server definitions in your configuration file. For details, see servopts(5) in the File Formats, Data Descriptions, MIBs, and System Processes Reference.)
5. Look for any core dumps in the directory defined by the APPDIR variable. Use a debugger such as dbx to get a stack trace. If you find core dumps, notify your application developer.
6. Check your system activity reports (for example, by running the sar(1) command) to determine why your system is not functioning properly. Consider the following reasons:
   • The system may be running out of memory.
   • The kernel might not be tuned correctly.

The following steps will help you detect the source of a system failure.

1. Check any Oracle Tuxedo system warnings and error messages in the user log ():
   • TPEOS messages indicate errors in the operating system.
   • TPESYSTEM messages indicate errors in the Oracle Tuxedo system.
2. Select the messages you think most likely reflect the current problem.

   Note:

   The catalog name and number of each of message, so you can look up the message in System Messages . The manual entry provides:

   • Details about the error condition flagged by the message.
   • Recommendations for recovery actions.
3. Prepare for debugging in the following ways:
   • Shut down the suspend service.
   • Use tmboot -n -s(server) -d1. (This will not boot the server, but prints the command line used to boot the server by the Oracle Tuxedo system.) Use that command line with a debugger such as dbx.

To print a UDL, complete the following procedure:

1. Run tmadmin -c
2. Enter the following command:

   lidl

3. To specify the device from which you want to obtain the UDL, you have a choice of two methods:
   • Specify the device on the lidl command line:

     -z device_name [devindx]

   • Set the environment variable FSCONFIG to the name of the desired device.

To print VTOC information, complete the following procedure.

1. Run tmadmin -c
2. To get information about all VTOC table entries, enter the following command:

   livtoc

3. To specify the device from which you want to obtain the VTOC, you have a choice of two methods:
   • Specify the following on the lidl command line:

     -z device name [devindx ]

   • Set the environment variable FSCONFIG to the name of the desired device.

To reinitialize a device that is included on a device list, complete the following procedure.

1. Run tmadmin -c
2. Enter the following command:
   initdl [-z devicename] [-yes] devindx

   Note:

   The value of devindx is the index to the file to be destroyed.
3. You can specify the device by:
   • Entering its name after the -z option (as shown here), or
   • Setting the environment variable FSCONFIG to the device name.
4. If you include the -yes option on the command line, you are not prompted to confirm your intention to destroy the file before the file is actually destroyed.

To create a device list, complete the following procedure.

1. Run tmadmin -c
2. Enter the following command:

   crdl [-z devicename] [-b blocks]

   • The value of devicename [devindx] is the desired device name. (Another way to assign a name to a new device is by setting the FSCONFIG environment variable to the desired device name.)
   • The value ofblocks is the number of blocks needed. The default is 1000 blocks.

   Note: Because 35 blocks are needed for the administrative overhead associated with a TLOG, be sure to assign a value higher than 35 when you create a TLOG.

To destroy a device list with index devindx , complete the following procedure.

1. Run tmadmin -c
2. Enter the following command:

   dsdl [-z devicename] [yes] [devindx]

   Note: The value of devindx is the index to the file to be destroyed.
3. You can specify the device by:
   • Entering its name after the -z option (as shown here), or
   • Setting the environment variable FSCONFIG to the device name
4. If you include the yes option on the command line, you are not prompted to confirm your intention to destroy the file before the file is actually destroyed.

You can detect a network partition in one of the following ways:

• Check the user log (ULOG) (ULOG) for messages that may shed light on the origin of the problem.
• Gather information about the network, server, and service, by running the tmadmin commands provided for this purpose.

• How to Check the ULOG
  

This topic provides instructions for recovering from transient and severe network failures.

• How to Recover from Transient Network Failures
  
• How to Recover from Severe Network Failures
  

To restore a failed MASTER machine, complete the following procedure.

1. Make sure that all IPC resources for the Oracle Tuxedo processes that are removed.
2. Start a tmadmin session on the ACTING MASTER (SITE2):

   tmadmin

3. Boot the BBL on the MASTER (SITE1) by entering the following command:

   boot -B SITE1

   (The BBL does not boot if you have not executed pclean on SITE1.)

4. Still in tmadmin, start a DBBL running again on the MASTER site (SITE1) by entering the following:

   MASTER

5. If you have migrated application servers and data off the failed machine, boot them or migrate them back.

To restore a failed nonmaster machine, complete the following procedure.

1. On the MASTER machine, start a tmadmin session
2. Run pclean, specifying the partitioned machine on the command line.
3. Fix the machine problem.
4. Restore the failed machine by booting the Bulletin Board Liaison (BBL) for the machine from the MASTER machine.

In the following list, SITE2, a nonmaster machine, is restored.

Listing 9‑3 Example of Restoring a Failed Nonmaster Machine

$ tmadmin 
tmadmin - .......................; ..................

> pclean SITE2 
Cleaning the DBBL. 

Pausing 10 seconds waiting for system to stabilize.
3 SITE2 servers removed from bulletin board

> boot -B SITE2 
Booting admin processes ... 

Exec BBL -A : 

on SITE2 -> process id=22923 ... Started. 
1 process started. 
> q

To request an immediate cleanup of resources associated with dead processes, complete the following procedure.

1. Start a tmadmin session.
2. Enter bbclean machine.

   The bbclean command takes one optional argument: the name of the machine to be cleaned.

To clean up other resources, complete the following procedure.

1. Start a tmadmin session.
2. Enter pclean machine .

   Note:

   You must specify a value for machine; it is a required argument.

This command is useful for restoring order to a system after partitioning has occurred unexpectedly.

If a Oracle Tuxedo CORBA application fails to boot, open the application’s UBBCONFIG file with a text editor and check whether the servers are booted in the correct order in the SERVERS section. The following is the correct order in which to boot the servers in a Oracle Tuxedo CORBA environment. A Oracle Tuxedo CORBA application will not boot if this order is not adhered to.

Boot the servers in the following order:

1. The system EventBroker, TMSYSEVT.
2. The TMFFNAME server with the -N option and the -M option, which starts the NameManager service (as a MASTER). This service maintains a mapping of application-supplied names to object references.
3. The TMFFNAME server with the -N option only, to start a slave NameManager service.
4. The TMFFNAME server with the -F option, to start the FactoryFinder.
5. The application servers that are advertising factories.

For a detailed example, see the section Required Order in Which to Boot CORBA C++ Servers in Setting Up an Oracle Tuxedo Application

The network address that is specified by programmers in the Bootstrap object constructor or in TOBJADDR must exactly match the network address in the server application’s UBBCONFIG file. The format of the address as well as the capitalization must match. If the addresses do not match, the call to the Bootstrap object constructor will fail with a seemingly unrelated error message:

For example, if the network address is specified as //TRIXIE:3500 in the ISL command-line option string (in the server application’s UBBCONFIG file), specifying either //192.12.4.6:3500 or //trixie:3500 in the Bootstrap object constructor or in TOBJADDR will cause the connection attempt to fail.

On UNIX systems, use the uname -n command on the host system to determine the capitalization used. On Windows systems, see the host system’s Network control panel to determine the capitalization used.

To abort a transaction, complete the following procedure.

1. Enter the following command:

   aborttrans (abort) [-yes] [-g groupname] tranindex

2. To determine the value of tranindex, run the printtrans command (a tmadmin command).
3. If groupname is specified, a message is sent to the TMS of that group to mark as aborted the transaction for that group. If a group is not specified, a message is sent, instead, to the coordinating TMS, requesting an abort of the transaction. You must send abort messages to all groups in the transaction to control the abort.

This command is useful when the coordinating site is partitioned or when the client terminates before calling a commit or an abort. If the timeout is large, the transaction remains in the transaction table unless it is aborted.

To commit a transaction, enter the following command:

committrans (commit) [-yes] [-g groupname] tranindex

The operation fails if the transaction is not precommitted or has been marked aborted. This message should be sent to all groups to fully commit the transaction.

• Cautions About Using the committrans Command
  

Multi-threaded applications can be much more difficult to debug than single-threaded applications. As the administrator, you may want to establish a policy governing whether such multithreaded applications should be created.

When running in protected mode, an application attaches to shared memory only when an ATMI call is being executed. Protected mode is used to guard against problems that arise when Oracle Tuxedo shared memory is accidentally overwritten by stray application pointers.

If your multithreaded application is running in protected mode, some threads may be executing application code while others are attached to the Oracle Tuxedo Bulletin Board’s shared memory within an Oracle Tuxedo function call. Therefore, as long as at least one thread is attached to the bulletin board in an ATMI call, the use of protected mode cannot guard against stray application pointers in threads executing application code, which may overwrite the Oracle Tuxedo shared memory. As a result, the usefulness of protected mode is relatively limited in multithreaded applications.

There is no solution to this limitation. We simply want to warn you that when running a multithreaded application you cannot rely on protected mode as much as you do when running a single-threaded application.