As an administrator, you must ensure that once an application is up and running, it continues to meet the performance, availability, and security requirements set by your company. To perform this task, you need to monitor the resources (such as shared memory), activities (such as transactions), and potential problems (such as security breaches) in your configuration, and take any necessary corrective actions.

To help you meet this responsibility, the Oracle Tuxedo system provides several methods for monitoring system and application events, and dynamically reconfiguring your system to improve performance. The following facilities offer an excellent view of how your system is working:

• Command-line utilities
• log files
• the ATMI
• the MIB
• run-time and user-level tracing facilities

These tools help make your application capable of responding quickly and efficiently to changing business needs or failure conditions. They also assist you in managing your application’s performance and security.

The following figure shows the monitoring tools.

Figure 2-1 Monitoring Tools

The Oracle Tuxedo system offers the following tools to monitor your application:

• Command-line utilities—a set of commands (for example, tmboot(1), tmadmin(1), and tmshutdown(1) you can use to activate, deactivate, configure, and manage your application.
• EventBroker—a mechanism that informs administrators of system faults and exceptional happenings such as network failures. When an event is posted by clients or servers, the EventBroker matches the name of the posted event to a list of subscribers for that event, and takes appropriate action, determined by each subscription.
• Log files—a set of files that make up a repository for error and warning messages, debugging messages, and informational messages helpful in tracking and resolving problems in the system.
• MIB—an interface to a set of procedures for accessing and modifying information in the MIBs. Using the MIB, you can write programs that enable you to monitor your run-time application.
• Run-time and User-level tracing facility—software that tracks the execution of an application, thus providing information that is helpful in resolving system problems

See Also

• System and Application Data That You Can Monitor
• Selecting Appropriate Monitoring Tools
• Using Command-line Utilities to Monitor Your Application
• Using EventBroker to Monitor Your Application
• Using Log Files to Monitor Activity
• Using ATMI to Handle System and Application Errors
• Using the MIB to Monitor Your Application
• Managing Operations Using the MIB in Introducing Oracle Tuxedo ATMI
• Using the Run-time and User-level Tracing Utility
• tmshutdown(1) in the Oracle Tuxedo Command Reference
• Oracle Tuxedo Management Tools in Introducing Oracle Tuxedo ATMI

The Oracle Tuxedo system enables you to monitor system and application data.

Monitoring System Data

To help you monitor a running system, your Oracle Tuxedo system maintains parameter settings and generates statistics for the following system components:

• Clients
• Conversations
• Groups
• Message queues
• Networks
• Servers
• Services
• CORBA Interfaces
• Transactions

You can access these components using the MIB or tmadmin. You can set up your system so that it can use the statistics in the bulletin board to make decisions and to modify system components dynamically, without your intervention. With proper configuration, your system can perform the following tasks (when bulletin board statistics indicate that they are required):

• Turn on load balancing
• Start a new copy of a server
• Shut down servers that are not being used

By monitoring the administrative data for your system, you can prevent and resolve problems that threaten the performance, availability, and security of your application.

Where the System Data Resides

To ensure that you have the information necessary to monitor your system, the Oracle Tuxedo system provides the following three data repositories:

• Bulletin board—a segment of shared memory (on each machine in your network) to which your system writes statistics about the components and activities of your configuration
• Log files—files to which your system writes messages
• UBBCONFIG—a text file in which you define the parameters of your system and application

• Monitoring Dynamic and Static Administrative Data
  

When evaluating whether your Oracle Tuxedo system is operating normally, you might want to consider the following list of common start-up and shutdown problems, and monitor your system periodically.

• Common Startup Problems
  
• Common Shutdown Problems
  

To monitor a running application, you need to keep track of the dynamic aspects of your configuration and sometimes check the static data. In other words, you need to be able to watch the bulletin board on an ongoing basis and consult the UBBCONFIG file when necessary. The method you choose depends on the following factors:

• Your Oracle Tuxedo system administration experience: If you have a lot of experience as an administrator, as well as shell programming expertise, you may prefer to write programs that automate your most frequently run commands.
• Which information you want to view: If you decide to monitor your application by examining the RESOURCES section of the UBBCONFIG file through the tmadmin command, you will have access to only the current values.

The following table describes how to use each monitoring method.

To monitor your application through the command-line interface, use the tmadmin(1) or txrpt(1) command.

• Inspecting Your Configuration Using tmadmin
  
• Generating Reports on Servers and Services Using txrpt
  

Thetmadmin command is an interpreter for 53 commands that enable you to view and modify a bulletin board and its associated entities. Figure2‑2 shows you how a typical tmadmin session works.

The following illustrates a Typical tmadmin Session

Figure 2-3 Typical tmadmin Session

• Monitoring Your System Using tmadmin Commands
  

The Oracle Tuxedo EventBroker monitors a running application for events (for example, a state change in a MIB object, such as the transition of a client from active to inactive). When the EventBroker detects an event, it reports or posts the event, and then notifies relevant subscribers that the event has occurred. You can be informed automatically when events occur in the MIB by receiving FML data buffers representing MIB objects. To post the event and report it to subscribers, the EventBroker uses the tppost(3c) function. Both administrators and application processes can subscribe to events.

The EventBroker recognizes over 100 meaningful state transitions to a MIB object as system events. A posting for a system event includes the current MIB representation of the object on which the event occurred, and some event-specific fields that identify the event that occurred. For example, if a machine is partitioned, an event is posted with the following:

• The name of the affected machine, as specified in the T_MACHINE class, with all the attributes of that machine
• Some event attributes identifying the event as machine partitioned

To use the EventBroker, you simply subscribe to system events.

See Also

• Managing Events Using EventBroker in Introducing Oracle Tuxedo ATMI

To help you identify error conditions quickly and accurately, the Oracle Tuxedo system provides the following log files:

• Transaction log (TLOG)—a binary file that is not normally read by you (the administrator), but that is used by the Transaction Manager Server (TMS). A TLOG is created only on machines involved in Oracle Tuxedo global transactions.
• User log (ULOG)—a log of messages generated by the Oracle Tuxedo system while your application is running. The ULOGMILLISEC environment variable is used to time stamp ulog message output intervals in milliseconds instead of seconds. The ULOGRTNSIZE environment variable is used to specify rotation files size. For more information on ULOGMILLISEC and ULOGRTNSIZE, see userlog(3c) in the Oracle Tuxedo Command Reference.

  These logs are maintained and updated constantly while your application is running.

The transaction log (TLOG) keeps track of global transactions during the commit phase. At the end of the first phase of a 2-phase commit protocol, the participants in a global transaction issue a reply to the question of whether to commit or roll back the transaction. This reply is recorded in the TLOG.

The TLOG file is used only by the Transaction Manager Server (TMS) that coordinates global transactions. It is not read by the administrator. The location and size of the TLOG are specified by four parameters that you set in the MACHINES section of the UBBCONFIG file.

You must create a TLOG on each machine that participates in global transactions.

See Also

Detecting Errors Using Logs

The user log (ULOG) is a file to which all messages generated by the Oracle Tuxedo system—error messages, warning messages, information messages, and debugging messages—are written. Application clients and servers can also write to the user log. A new log is created every day and there can be a different log on each machine. However, a ULOG can be shared by multiple machines when a remote file system is being used.

The ULOG provides an administrator with a record of system events from which the causes of most Oracle Tuxedo system and application failures can be determined. You can view the ULOG, a text file, with any text editor. The ULOG also contains messages generated by the tlisten process. The tlisten process provides remote service connections for other machines in an application. Each machine, including the master machine, should have a tlisten process running on it.

The Oracle Tuxedo log files can help you detect failures in both your application and your system by:

• Analyzing the Transaction Log (TLOG)
  
• Analyzing the User Log (ULOG)
  
• Analyzing tlisten Messages in the ULOG
  

A Oracle Tuxedo application server can generate a log of the service requests it handles. The log is displayed on the server’s standard output (stdout). Each record contains a service name, start time, and end time.

You can request such a log when a server is activated. The txrpt facility produces a summary of the time spent by the server, thus giving you a way to analyze the log output. Using this data, you can estimate the relative workload generated by each service, which will help you set workload parameters appropriately for the corresponding services in the MIB.

ud32 is a client program delivered with the Oracle Tuxedo system that reads input consisting of text representation of FML buffers. You can use ud32 for adhoc queries and updates to the MIB. It creates an FML32 buffer, makes a service call with the buffer, receives a reply (also in an FML32 buffer) from the service call, and displays the results on screen or in a file in text format.

ud32 builds an FML32-type buffer with the FML fields and values that you represent in text format, makes a service call to the identified service in the buffer, and waits for the reply. The reply then comes back in FML32 format as a report. Now, because the MIB is FML32-based, ud32 becomes the scripting tool for the MIB.

For example, suppose you write a small file that contains the following text:

SRVCNM .TMIB

TA_OPERATION GET

TA_CLASS T_SERVER

The Oracle Tuxedo system provides a run-time and user-level tracing facility that enable you to track the execution of distributed business applications. The system has a set of built-in trace points that mark calls to functions in different categories, such as ATMI functions issued by the application or XA functions issued by the Oracle Tuxedo system to an X/Open compliant resource manager.

To enable tracing, you must specify a tracing expression that contains a category, a filtering expression, and an action. The category indicates the type of function (such as ATMI) to be traced. The filtering expression specifies which particular functions trigger an action. The action indicates the response to the specified functions by the Oracle Tuxedo system.

The system may, for example, write a record in the ULOG, execute a system command, or terminate a trace process. A client process can also propagate the tracing facility with its requests. This capability is called dyeing; the trace dye colors all services that are called by the client.

• Specifying the expression in a server environment

  You can also specify a tracing expression in the environment of a server using the ulog or utrace tmtrace(5) receivers. For example, you might enter the following:

  • Run-time Tracing Expression: TMTRACE=atmi:/tpservice/ulog. If you export this setting within a server environment, a record with general run-time trace information is created in the ULOG file for all service requests performed on that server.
  • User-Level Expression: TMTRACE=atmi:utrace. Specifying the utrace receiver automatically calls the user-defined tputrace(3c). If you export this setting within a server environment, a record with trace information and output location defined by the user is created for the ATMI functions running on that server.

You can activate or deactivate the tracing option using the changetrace command of tmadmin. This command enables you to overwrite the tracing expression on active client or server processes. Administrators can enable global tracing for all clients and servers, or for a particular machine, group, or server.

See Also

• Ways to Monitor Your Application
• tmtrace(5) in File Formats, Data Descriptions, MIBs, and System Processes Reference
• userlog(3c) and tputrace(3c) in Oracle Tuxedo ATMI C Function Reference

There are essentially two operations you can perform using the MIB: you can get information from the MIB (a get operation) or you can update information in the MIB (a set operation) at any time using a set of ATMI functions (for example, tpalloc(3c), tprealloc(3c), tpcall(3c), tpacall(3c),tpgetrply(3c),tpenqueue(3c) ).

When you query the MIB with a get operation, the MIB responds to your reply with a number of matches, and indicates how many more objects match your request. The MIB returns a handle (that is, the cursor) that you can use to get the remaining objects. The operation you use to get the next set of objects is called getnext. The third operation occurs when queries span multiple buffers.

• Limiting Your MIB Queries
  
• Querying Global and Local Data
  
• Using tmadmcall to Access Information
  

The Oracle Tuxedo system uses the following two administrative servers to distribute the information on the bulletin board to all active machines in the application:

• DBBL—the Distinguished Bulletin Board Liaison server propagates global changes to the MIB and maintains the static part of the MIB. Specifically, the DBBL:
  • Resides (only one DBBL per application) on the MASTER machine and provides periodic status requests to all BBLs
  • Coordinates migration of servers
  • Can be migrated to other machines for fault resiliency
• BBL—the Bulletin Board Liaison server maintains the bulletin board on its host machine, coordinating changes to the local MIB, and verifying the integrity of application programs active on its machine. Specifically, the bulletin board:
  • Resides on each Oracle Tuxedo machine in an application, carries out requests from the DBBL, and administers timeouts for service requests, replies to requesters, and transactions
  • Detects server failures, initiates user-defined recovery, and automatically restarts servers
  • Detects client failures
  • Cleans up client and server entries, and conversations on the bulletin board
  • Detects and recovers DBBL failures (if it is the BBL residing on the MASTER machine)

The following figure illustrates the diagnosis and repair using the DBBL and BBLs.

Figure 2-4 Diagnosis and Repair Using the DBBL and BBLs

Both servers have a role in managing faults. The DBBL coordinates the state of other active machines in the application. Each BBL communicates state changes in the MIB, and sometimes sends a message to the DBBL indicating all is OK on its host machine.

The Oracle Tuxedo run-time system records events, along with system errors, warnings, and tracing events, in the user log (ULOG). Programmers can use the ULOG to debug their applications or notify administrators of special conditions or states found (for example, an authorization failure).

Using ATMI, a programmer controls some of the more global aspects of communications. ATMI provides functions for handling both application and system-related errors. When a service routine encounters an application error, such as an invalid account number, the client knows the service performed its task but could not fulfill its request because of an application error.

With a system failure, such as a server crashing while performing a request, the client knows the service routine did not perform its task because of an underlying system error. The Oracle Tuxedo system notifies programs of system errors that occur as it monitors the application’s behavior and its own behavior.

• Using Configurable Timeout Mechanisms
  
• Configuring Redundant Servers to Handle Failures
  

While monitoring a multithreaded application, keep in mind that individual threads are not visible to an administrator.

• While monitoring a multithreaded application, keep in mind that individual threads are not visible to an administrator.
• You can get MIB statistical reports for various aspects of your multithreaded and/or multicontexted application by running the tmadmin(1) command interpreter. Here are a few examples of the information you can request for a multithreaded application:
  • Count of dispatched services per server process and, optionally, information about each context (obtained by running tmadmin/psr, optionally in verbose mode).
  • Count of client contexts per client process and a separate entry for each client context (obtained by running the tmadmin pclt command).
• When the BBL checks clients, it verifies that a process is alive. If a process has died, the BBL detects the process death. If an individual thread within a process has died, however, the death of the thread is not detected by the BBL.

Therefore application programmers should keep in mind the possibility that individual threads within a process may die. If one thread dies and a signal is issued, the whole process to which the thread belongs usually dies, and that death is detected by the BBL.

If a thread dies as the result of an erroneous call to a thread exit function, however, no signal is generated. If this type of death occurs before the thread calls tpterm(), then the BBL cannot detect the death and does not deallocate the registry table slot for the context associated with the dead thread. (It would not be proper for the BBL to deallocate this registry table slot even if it could detect the death of the thread because, in some application models, another thread might subsequently choose to associate itself with that context.)

There is no solution for this limitation so it is important for programmers to keep it in mind and design their applications accordingly.

• How to Retrieve Data About a Multithreaded/Multicontexted Application Using the MIB