11 IPC Resource Configuration on a UNIX System
The following sections describe the inter-process communication (IPC) parameters on a UNIX system and provide guidelines for configuring them:
11.1 Parameter Sets Controlling IPC Resources
On a UNIX system, the Oracle Tuxedo system uses the IPC resources provided by the UNIX operating system, which are controlled by the tunable parameters listed in the following table:
Table 11-1 Tunable Parameters
Prefix of Tunable Parameters | Resources Controlled |
---|---|
SHM |
Amount of shared memory |
SEM |
Number of semaphores |
MSG |
Size of message queues and messages |
The settings for these parameters are application-dependent. Most UNIX systems are shipped with default values that are too low for an Oracle Tuxedo application.
Because the IPC parameters vary across different versions of the UNIX system, the descriptions provided in the following sections are generic. For the exact parameter names and defaults for each platform and for information on how to change parameter values, see Oracle Tuxedo 22c Release (22.1.0.0.0) Platform Data Sheets. If you change a parameter value, then must rebuild the kernel and reboot the operating system, using standard administrative tools. Consult your operating system administrator or the system administrator’s guide for your platform for details.
If your Oracle Tuxedo application is distributed, the minimum IPC resources must be available on every UNIX platform participating in the application.
- Shared Memory
- Semaphores
- Message Queues and Messages
- What Is Application Deadlock?
- Performance Implications of Blocking Conditions
- Tunable Message Parameters
- Other Kernel Tunables
Parent topic: IPC Resource Configuration on a UNIX System
11.1.1 Shared Memory
In the Oracle Tuxedo environment, shared memory is used for the bulletin board and the control table of the workstation listener (WSL) and the IIOP listener (ISL) processes. An application may also use shared memory for its own purposes. The following shared memory parameters may need to be adjusted:
-
SHMMAX
- Maximum size, in bytes, of a shared memory segment. This number represents the largest shared memory segment that can be allocated. A process can, however, attach to more than one segment of size
SHMMAX
. -
SHMSEG
- Maximum number of shared memory segments per process. For a given configuration, the maximum amount of shared memory to which a process can attach is the product (in bytes) of
SHMMAX * SHMSEG
. A value between 6 and 15 should be adequate. -
SHMMNI
- Maximum number of shared memory identifiers in the system. The Oracle Tuxedo system requires one identifier per bulletin board and an additional identifier for each workstation listener (WSL) and IIOP listener (ISL) that is running.
-
SHMMIN
- Minimum size, in bytes, of shared memory segment. This parameter should always be set to 1.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.2 Semaphores
Every process that participates in an Oracle Tuxedo application requires a semaphore. A semaphore is a hardware or software flag used to prevent processes from accessing the same shared memory space at the same time. When a process has control of a shared memory resource, all other processes are locked out of the shared memory resource until the process releases the resource. When the Oracle Tuxedo application is booted, the underlying Oracle Tuxedo system checks the number of semaphores configured in the operating system. If the configured number is not high enough, the boot fails.
The following semaphore parameters may need to be adjusted:
-
SEMMNS
- Maximum number of semaphores in the system. The minimum requirement for
SEMMNS
isMAXACCESSERS - MAXWSCLIENTS + 13
, whereMAXACCESSERS
is the maximum number of Oracle Tuxedo system processes on a particular machine (including servers and native clients) andMAXWSCLIENTS
is the maximum number of Oracle Tuxedo remote clients. Both of these parameters are specified in theUBBCONFIG
file for the application. For more information aboutUBBCONFIG
, see Creating the Configuration File orUBBCONFIG(5)
in the File Formats, Data Descriptions, MIBs, and System Processes Reference. -
SEMMNI
- Maximum number of active semaphore sets.
-
SEMMSL
- Maximum number of semaphores per semaphore set.
SEMMNI
andSEMMSL
are commonly chosen so that their product equalsSEMMNS
. The Oracle Tuxedo system does not perform semaphore operations on semaphore sets; however, it attempts to allocate as many semaphores per semaphore set as possible. -
SEMMAP
- Size of the control map used to manage semaphore sets.
SEMMAP
should be equal toSEMMNI
. -
SEMMNU
- Number of undo structures in the system. Because an undo structure is needed for each process that can access the bulletin board,
SEMMNU
must be at least as large asSEMMNS
. (The UNIX operating system uses undo structures to unlock semaphores held by processes that die unexpectedly.) -
SEMUME
- Maximum number of undo entries per undo structure. The value 1 suffices.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.3 Message Queues and Messages
The Oracle Tuxedo system uses UNIX system messages and message queues for client/server communication. Examples of such messages are service requests, service replies, conversational messages, unsolicited notification messages, administrative messages, and transaction control messages.
Every Multiple Servers, Single Queue (MSSQ) set of servers and every individual server has a message queue for receiving requests. Every client has its own queue for receiving replies. Servers that specify theREPLYQ
parameter also get individual reply queues. The adjustment of kernel message parameters is important to the proper tuning of an application. Inappropriate values can lead to an inability to boot, or to severe performance degradation.
Several message queue parameters are available to define various characteristics of the queue space, as shown in the table below:
Table 11-2 Message Queue Parameters
Parameter | Description |
---|---|
MSGTQL |
Total number of outstanding messages that can be stored by the kernel. |
MSGMNB |
Total number of bytes that can be stored on one queue |
MSGMAX |
Maximum size of an individual message |
MSGSEG |
Total number of message segments that can be outstanding at one time |
MSGSSZ |
Size of each segment |
If the limit specified by any of these parameters is exceeded, then a blocking condition occurs. There is one exception to this rule: MSGMAX
. Messages that exceed 75 percent of MSGMNB
, or that are larger than MSGMAX
, are placed in a UNIX file. A very small message containing the file name is then sent to the recipient. Because this mode of operation results in a severe reduction in performance, we strongly recommend that you avoid it.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.4 What Is Application Deadlock?
An application deadlock can result if every process is blocked while trying to send a message. For example, when clients fill up the message space with requests, servers that are trying to send replies are blocked. Therefore, no server can read a message and a deadlock results. Occasionally, timeouts can break a deadlock, but no useful work will have been done.
A client that sends its requests with the TPNOREPLY
flag is especially troublesome. This practice can fill either individual queues or the system message space, depending on the size of the messages. Such applications may have to implement their own flow control to limit the number of outstanding messages. To summarize, if clients or servers are blocking on their send operations (requesting services or sending replies), there is potential for trouble. It is usually no problem, though, for a single server request queue to remain full, as long as there is space in the system for more messages on other queues.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.5 Performance Implications of Blocking Conditions
There are performance implications to queue blocking conditions, both on the sending side and the receiving side. When waking up blocked processes, the UNIX operating system wakes up all the processes blocked on a particular event, even if only one can proceed. The other processes go back to sleep. This process scheduling overhead can be expensive.
For example, on an empty server request queue on which more than one server (MSSQ) resides, an arriving message wakes up all the idle (blocked) servers on that queue. In the case of a full server request queue, as each request is read by a server, the system wakes up all the blocked clients. Depending on the size of the messages, zero or more clients can place messages on the queue. The rest go back to sleep. Because there may be hundreds of clients in the system, the mass wakeup of all of these clients every time a service request is processed can severely degrade performance.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.6 Tunable Message Parameters
A properly tuned system rarely fills its queues. Enough slack should be left in the queues to handle the natural variability of the message flow. No exact settings can be recommended. Tuning is very application dependent. The UNIX ipcs(1)
command provides a snapshot of the queues so you can determine whether they are full. You can try setting the TPNOBLOCK
flag when sending requests. If you do, clients can tell when queues are full, and they can slow down a bit. It might help to increase the scheduling priority of servers with full request queues.
The following message parameters may need to be adjusted:
-
MSGMNI
- Number of unique message queue identifiers. Each process participating in an Oracle Tuxedo application on a particular machine typically needs at least one message queue. This number is reduced if MSSQ sets are used, which means that multiple server processes share a single queue. For transaction processing, count an additional queue per server group for transaction manager server (TMS) processes. Thus, the minimum requirement for
MSGMNI
can be determined by the following formula: -
MSGMAX
- Maximum message size in bytes.
MSGMAX
must be big enough to handle any Oracle Tuxedo application running on this machine. -
MSGMNB
- Maximum message queue length in bytes. This number must accommodate the total size of all messages that are on a queue and have not been taken off by the associated processes. The minimum value for
MSGMNB
is the value ofMSGMAX
. Messages longer than 75% ofMSGMNB
are sent to a file instead of a message queue—a situation that should be avoided because it severely degrades performance. -
MSGMAP
- Number of entries in the control map used to manage message segments. The value of
MSGMAP
should be the number of message segments (specified inMSGSEG
). - MSGSSZ
- Size, in bytes, of a message segment. A message can consist of several such segments. The value of
MSGSSZ
should be such that a multiple ofMSGSSZ
is equal to the size (including the Oracle Tuxedo system header) of the most commonly sent message. By dividing messages into segments in this way, you can avoid wasting space. -
MSGSEG
- Number of message segments in the system.
-
MSGTQL
- Total number of outstanding messages that can be stored by the kernel. This is the maximum number of unread messages at any given time.
Parent topic: Parameter Sets Controlling IPC Resources
11.1.7 Other Kernel Tunables
Experience with the Oracle Tuxedo system has shown that some other UNIX system tunables may need to be set to higher values. These parameters are very application dependent and do not apply to all applications. Oracle Tuxedo 22c Release (22.1.0.0.0) Platform Data Sheets includes information on the defaults for each platform and instructions for changing them.
-
ULIMIT
- Maximum file size.
ULIMIT
needs to be large enough so that you can install the Oracle Tuxedo system and build servers. We recommend 4 MB. -
NOFILES
- Maximum number of open files per process. An Oracle Tuxedo server requires a minimum of four file descriptors.
-
MAXUP
- Maximum number of processes per non-superuser. The Oracle Tuxedo system processes—servers and administrative processes—run with the UID specified in the application’s
UBBCONFIG
file.MAXUP
needs to be large enough to allow all of these processes to run. -
NPROC
- Maximum number of processes (system wide).
NREGION
Number of region table entries to allocate. Most processes have three regions: text, data, and stack. Additional regions are needed for each shared memory segment and each shared library (including text and data) that is attached. However, the region table entry for the text of a “shared text” program is shared by all processes executing that program. Each shared memory segment attached to one or more processes uses another region table entry. -
NUMTIM
- Maximum number of STREAMS modules that can be pushed by the Transport Layer Interface (TLI). A typical default value is 16; we recommend setting this parameter to at least 256.
-
NUMTRW
- The number of TLI read/write structures to allocate in kernel data space. A typical default value is 16; we recommend setting this parameter to at least 256.
Parent topic: Parameter Sets Controlling IPC Resources