5 Monitoring and Troubleshooting

5.1.1 Failure Prevention and Automatic Recovery Features

Oracle WebLogic Server SIP Container, and the underlying WebLogic Server platform, provide many features that protect against server failures. In a production system, all available features should be used in order to ensure uninterrupted service.

5.1.2 Directory and File Backups for Failure Recovery

Recovery from the failure of a server instance requires access to the domain's configuration data. By default, the Administration Server stores a domain's primary configuration data in a file called domain_name/config/config.xml, where domain_name is the root directory of the domain. The primary configuration file may reference additional configuration files for specific WebLogic Server services, such as JDBC and JMS, and for Oracle WebLogic Server SIP Container services, such as SIP container properties and SIP data tier configuration. The configuration for specific services are stored in additional XML files in subdirectories of the domain_name/config directory, such as domain_name/config/jms, domain_name/config/jdbc, and domain_name/config/custom for Oracle WebLogic Server SIP Container configuration files.

The Administration Server can automatically archive multiple versions of the domain configuration (the entire domain-name/config directory). The configuration archives can be used for system restoration in cases where accidental configuration changes need to be reversed. For example, if an administrator accidentally removes a configured resource, the prior configuration can be restored by using the last automated backup.

The Administration Server stores only a finite number of automated backups locally in domain_name/config. For this reason, automated domain backups are limited in their ability to guard against data corruption, such as a failed hard disk. Automated backups also do not preserve certain configuration data that are required for full domain restoration, such as LDAP repository data and server start-up scripts. Oracle recommends that you also maintain multiple backup copies of the configuration and security offline, in a source control system.

This section describes file backups that Oracle WebLogic Server SIP Container performs automatically, as well as manual backup procedures that an administrator should perform periodically.

cd ~/user_projects/domains/mydomain
tar cvf domain-backup-06-17-2007.jar config

domain_name\adminServer\ldap

5.1.3 Restarting a Failed Administration Server

When you restart a failed Administration Server, no special steps are required. Start the Administration Server as you normally would.

If the Administration Server shuts down while Managed Servers continue to run, you do not need to restart the Managed Servers that are already running in order to recover management of the domain. The procedure for recovering management of an active domain depends upon whether you can restart the Administration Server on the same machine it was running on when the domain was started.

5.1.4 Restarting Failed Managed Servers

If the machine on which the failed Managed Server runs can contact the Administration Server for the domain, simply restart the Managed Server manually or automatically using Node Manager. Note that you must configure Node Manager and the Managed Server to support automated restarts.

If the Managed Server cannot connect to the Administration Server during startup, it can retrieve its configuration by reading locally-cached configuration data. A Managed Server that starts in this way is running in Managed Server Independence (MSI) mode.

To start up a Managed Server in MSI mode:

1. Ensure that the following files are available in the Managed Server's root directory:

   • msi-config.xml

   • SerializedSystemIni.dat

   • boot.properties

   If these files are not in the Managed Server's root directory:

   1. Copy the config.xml and SerializedSystemIni.dat file from the Administration Server's root directory (or from a backup) to the Managed Server's root directory.

   2. Rename the configuration file to msi-config.xml. When you start the server, it will use the copied configuration files.

      Note:

      Alternatively, use the -Dweblogic.RootDirectory=path startup option to specify a root directory that already contains these files.

2. Start the Managed Server at the command line or using a script.

   The Managed Server will run in MSI mode until it is contacted by its Administration Server. For information about restarting the Administration Server in this scenario, see Section 5.1.3, "Restarting a Failed Administration Server".

5.2.1 WlssEchoServer Failure Detection

WlssEchoServer is a separate process that you can run on the same server hardware as a SIP data tier replica. The purpose of WlssEchoServer is to provide a simple UDP echo service to engine tier nodes to be used for determining when a SIP data tier server goes offline, for example in the event that the network cable is disconnected. The algorithm for detecting failures with WlssEchoServer is as follows:

1. For all normal traffic, engine tier servers communicate with SIP data tier replicas using TCP. TCP is used as the basic transport between the engine tier and SIP data tier regardless of whether or not WlssEchoServer is used.

2. Engine tier servers send a periodic heartbeat message to each configured WlssEchoServer over UDP. During normal operation, WlssEchoServer responds to the heartbeats so that the connection between the engine node and replica is verified.

3. Should there be a complete failure of the SIP data tier stack, or the network cable is disconnected, the heartbeat messages are not returned to the engine node. In this case, the engine node can mark the replica as being offline without having to wait for the normal TCP connection timeout.

4. After identifying the offline server, the engine node reports the failure to an available SIP data tier replica, and the SIP data tier view is updated as described in the previous section.

Also, should a SIP data tier server notice that its local WlssEchoServer process has died, it automatically shuts down. This behavior ensures even quicker failover because avoids the time it takes engine nodes to notice and report the failure as described in Section 5.2, "Overview of Failover Detection".

You can configure the heartbeat mechanism on engine tier servers to increase the performance of failover detection as necessary. You can also configure the listen port and log file that WlssEchoServer uses on SIP data tier servers.

5.2.2 Forced Shutdown for Failed Replicas

If any engine tier server cannot communicate with a particular replica, the engine access another, available replica in the SIP data tier to report the offline server. The replica updates its view of the affected partition to remove the offline server. The updated view is then distributed to all engine tier servers that later access the partition. Propagating the view in this manner helps to ensure that engine servers do not attempt to access the offline replica.

The replica that updates the view also issues a one-time request to the offline replica to ask it to shut down. This is done to try to shut-down running replica servers that cannot be accessed by one or more engine servers due to a network outage. If an active replica can reach the replica marked as "offline," the offline replica shuts down.

5.3.1 Starting WlssEchoServer on SIP Data Tier Server Machines

WlssEchoServer is a Java program that you can start directly from a shell or command prompt. The basic syntax for starting WlssEchoServer is:

java -classpath WLSS_HOME/server/lib/wlss/wlssechosvr.jar options com.bea.wcp.util.WlssEchoServer

Where WLSS_HOME is the path to the Oracle WebLogic Server SIP Container installation and options may include one of the options described in Table 5-1.

-Dwlss.ha.echoserver.ipaddress

-Dwlss.ha.echoserver.port

-Dwlss.ha.echoserver.logfile

Oracle recommends that you include the command to start WlssEchoServer in the same script you use to start each Oracle WebLogic Server SIP Container SIP data tier instance. If you use the startManagedWebLogic.sh script to start an engine or SIP data tier server instance, add a command to start WlssEchoServer before the final command used to start the server. For example, change the lines:

"$JAVA_HOME/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS}     \
  -Dweblogic.Name=${SERVER_NAME}                                 \
  -Dweblogic.management.username=${WLS_USER}                     \
  -Dweblogic.management.password=${WLS_PW}                       \
  -Dweblogic.management.server=${ADMIN_URL}                      \
  -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \
   weblogic.Server

to read:

"$JAVA_HOME/bin/java" -classpath WLSS_HOME/server/lib/wlss/wlssechosvr.jar    \
  -Dwlss.ha.echoserver.ipaddress=192.168.1.4                   \
  -Dwlss.ha.echoserver.port=6734 com.bea.wcp.util.WlssEchoServer &
"$JAVA_HOME/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS}     \
  -Dweblogic.Name=${SERVER_NAME}                                 \
  -Dweblogic.management.username=${WLS_USER}                     \
  -Dweblogic.management.password=${WLS_PW}                       \
  -Dweblogic.management.server=${ADMIN_URL}                      \
  -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \
   weblogic.Server 

5.3.2 Enabling and Configuring the Heartbeat Mechanism on Servers

To enable the WlssEchoServer heartbeat mechanism, you must include the -Dreplica.host.monitor.enabled JVM argument in the command you use to start all engine and SIP data tier servers. Oracle recommends adding this option directly to the script used to start Managed Servers in your system. For example, in the startManagedWebLogic.sh script, change the line:

# JAVA_OPTIONS="-Dweblogic.attribute=value -Djava.attribute=value"

to read:

JAVA_OPTIONS="-Dreplica.host.monitor.enabled=true"

Several additional JVM options configure the functioning of the heartbeat mechanism. Table 5-1 describes the options used to configure failure detection.

-Dreplica.host.monitor.enabled

-Dwlss.ha.heartbeat.interval

-Dwlss.ha.heartbeat.count

-Dwlss.ha.heartbeat.SoTimeout

5.4.1 Browsing the MIB

The Oracle WebLogic Server SIP Container MIB file is installed in WLSS_HOME/server/lib/wlss/BEA-WLSS-MIB.asn1. Use an available SNMP management tool or MIB browser to view the contents of this file. See also Section 5.5.2, "Trap Descriptions" for a description of common SNMP traps.

5.4.2 Steps for Configuring SNMP

To enable SNMP monitoring for the entire Oracle WebLogic Server SIP Container domain, follow these steps:

1. Login to the Administration Console for the Oracle WebLogic Server SIP Container domain.

2. In the left pane, select the Diagnostics > SNMP node.

3. In the Server SNMP Agents table, click the New button to create a new agent.

   Note:

   Ensure that you create a new Server SNMP agent, rather than a Domain-Scoped agent.

4. Enter a unique name for the new SNMP agent (for example, "engine1snmp") and click OK.

5. Select the newly-created SNMP agent from the Server SNMP Agents table.

6. On the Configuration > General tab:

   1. Select the Enabled check box to enable the agent.

   2. Enter an unused port number in the SNMP UDP Port field.

      Note:

      If you run multiple Managed Server instances on the same machine, each server instance must use a dedicated SNMP agent with a unique SNMP port number.

   3. Click Save.

7. Repeat the above steps to generate a unique SNMP agent for each server in your deployment (SIP data tier server, engine tier server, and Administration Server).

5.5.1 Files for Troubleshooting

The following Oracle WebLogic Server SIP Container log and configuration files are frequently helpful for troubleshooting problems, and may be required by your technical support contact:

• $DOMAIN_DIR/config/config.xml

• $DOMAIN_DIR/config/custom/sipserver.xml

• $DOMAIN_DIR/servername/*.log (server and message logs)

• sip.xml (in the /WEB-INF subdirectory of the application)

• web.xml (in the /WEB-INF subdirectory of the application)

General information that can help the technical support team includes:

• The specific versions of:

  • Oracle WebLogic Server SIP Container

  • Java SDK

  • Operating System

• Thread dumps for hung Oracle WebLogic Server SIP Container processes

• Network analyzer logs

5.5.2 Trap Descriptions

Table 5-3 lists the Oracle WebLogic Server SIP Container SNMP traps and indicates whether the trap is generated by servers in the engine tier or SIP data tier. Each trap is described in the sections that follow.

5.6.1 Data Collection and Logging

Oracle WebLogic Server SIP Container uses the WLDF Harvester service to collect data from the attributes of these runtime MBeans:

• ReplicaRuntimeMBean

• SipApplicationRuntimeMBean

• SipServerRuntimeMBean

You can add charts and graphs of this data to your own custom views using the WLDF console extension. To do so, first enable the WLDF console extension by copying the JAR file into the console-ext subdirectory of your domain directory:

cp ~/bea/wlserver_10.3/server/lib/console-ext/diagnostics-console-extension.jar ~/bea/user_projects/domains/mydomain/console-ext

When accessing the WLDF console extension, the Oracle WebLogic Server SIP Container runtime MBean attributes are available in the Metrics tab of the extension.

Oracle WebLogic Server SIP Container also uses the WLDF Logger service to archive SIP and Diameter messages to independent, dedicated log files (by default, domain_home/logs/server_name/sipMessages.log). You can configure the name and location of the log file, as well as log rotation policies, using the Configuration > Message Debug tab in the SIP Server Administration Console extension. Note that a server restart is necessary in order to initiate independent logging and log rotation.

5.6.2 Watches and Notifications

The data collected from Oracle WebLogic Server SIP Container runtime MBeans can be used to create automated monitors, or "watches," that observe a server's diagnostic state. One or more notifications can then be configured for use by a watch, in order to generate a message using SMTP, SNMP, JMX, or JMS when your configured watch conditions and rules occur.

To use watches and notifications, you select the Diagnostics > Diagnostic Modules node in the left pane of the Administration Console and create a new module with the watch rules and notifications required for monitoring your servers. The watch rules can use the metrics collected from Oracle WebLogic Server SIP Container runtime MBeans, messages written to the log file, or events generated by the diagnostic framework.

5.6.3 Image Capture

Oracle WebLogic Server SIP Container adds its own image capture information to the diagnostic image generated by the WLDF. You can generate diagnostic images either on demand, or automatically by configuring watch rules.

The information contained in diagnostic images is intended for use by Oracle technical support personnel when troubleshooting a potential server problem and includes:

• SIP data tier partition and replica configuration

• Call state and timer statistics

• Work manager statistics

5.6.4 Instrumentation

The WLDF instrumentation system creates diagnostic monitors and inserts them into Oracle WebLogic Server SIP Container or application code at specific points in the flow of execution. Oracle WebLogic Server SIP Container integrates with the instrumentation service to provide a built-in DyeInjection monitor. When enabled, this monitor injects dye flags into the diagnostic context when certain SIP messages enter or exist the system. Dye flags are injected based on the monitor's configuration properties, and on certain request attributes.

Oracle WebLogic Server SIP Container adds the dye flags described in Table 5-4 below, as well as the WebLogic Server dye flags USER and ADDR. See Oracle Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server for more information.

Dye flags can be applied to both incoming and outbound SIP messages. The flags are useful for dye filtering, and can be used by delegating monitors to trigger further diagnostic actions.

Oracle WebLogic Server SIP Container provides several delegating monitors that can be applied at the application and server scope, and which may examine dye flags set by the DyeInjection monitor. The delegating monitors are described in Table 5-4.

<wldf-resource xmlns="http://www.bea.com/ns/weblogic/90/diagnostics">
  <instrumentation>
    <enabled>true</enabled>
    <include>demo.ProxyServlet</include>
    <wldf-instrumentation-monitor>
      <name>occas/Sip_Servlet_Before_Service</name>
      <enabled>true</enabled>
      <dye-mask>SIP_ANY_HEADER</dye-mask>
      <dye-filtering-enabled>true</dye-filtering-enabled>
      <action>DisplayArgumentsAction</action>
    </wldf-instrumentation-monitor>  
   </instrumentation>
</wldf-resource>

5.7.1 Defining Logging Servlets in sip.xml

Logging Servlets for SIP messages are created by defining Servlets having the implementation class com.bea.wcp.sip.engine.tracing.listener.TraceMessageListenerImpl. The definition for a sample msgTraceLogger is shown in Example 5-2.

<servlet>
    <servlet-name>msgTraceLogger</servlet-name>
    <servlet-class>com.bea.wcp.sip.engine.tracing.listener.TraceMessageListenerImpl</servlet-class>
    <init-param>
      <param-name>domain</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>level</param-name>
      <param-value>full</param-value>
    </init-param>
    <load-on-startup/>
  </servlet>

5.7.2 Configuring the Logging Level and Destination

Logging attributes such as the level of logging detail and the destination log file for SIP messages are passed as initialization parameters to the logging Servlet. Table 5-5 lists the parameters and parameter values that you can specify as init-param entries. Example 5-2 shows the sample init-param entries for a Servlet that logs full SIP message information to the domain log file.

5.7.3 Specifying the Criteria for Logging Messages

The criteria for selecting SIP messages to log can be defined either in XML files that are packaged with the logging Servlet's application, or as initialization parameters in the Servlet's sip.xml deployment descriptor. The sections that follow describe each method.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE pattern
   PUBLIC "Registration//Organization//Type Label//Definition Language"
   "trace-pattern.dtd">
<pattern>
  <equal>
    <var>response.method</var>
    <value>MESSAGE</value>
  </equal>
</pattern>

<servlet>
      <servlet-name>invTraceLogger</servlet-name>
      <servlet-class>com.bea.wcp.sip.engine.tracing.listener.TraceMessageListenerImpl</servlet-class>
      <init-param>
        <param-name>domain</param-name>
        <param-value>true</param-value>
      </init-param>
      <init-param>
        <param-name>level</param-name>
        <param-value>full</param-value>
      </init-param>
      <init-param>
        <param-name>request-pattern-string</param-name>
        <param-value>
            <![CDATA[
                <?xml version="1.0" encoding="UTF-8"?>
                <!DOCTYPE pattern
                   PUBLIC "Registration//Organization//Type Label//Definition Language"
                   "trace-pattern.dtd">
                <pattern>
                  <equal>
                    <var>request.method</var>
                    <value>INVITE</value>
                  </equal>
                </pattern>
            ]]>
        </param-value>
      </init-param>
      <init-param>
        <param-name>response-pattern-string</param-name>
        <param-value>
            <![CDATA[
                <?xml version="1.0" encoding="UTF-8"?>
                <!DOCTYPE pattern
                   PUBLIC "Registration//Organization//Type Label//Definition Language"
                   "trace-pattern.dtd">
                <pattern>
                  <equal>
                    <var>response.method</var>
                    <value>OPTIONS</value>
                  </equal>
                </pattern>
            ]]>
        </param-value>
      </init-param>
      <load-on-startup/>
  </servlet>

5.7.4 Specifying Content Types for Unencrypted Logging

By default Oracle WebLogic Server SIP Container uses String format (UTF-8 encoding) to log the content of SIP messages having a text or application/sdp Content-Type value. For all other Content-Type values, Oracle WebLogic Server SIP Container attempts to log the message content using the character set specified in the charset parameter of the message, if one is specified. If no charset parameter is specified, or if the charset value is invalid or unsupported, Oracle WebLogic Server SIP Container uses Base-64 encoding to encrypt the message content before logging the message.

If you want to avoid encrypting the content of messages under these circumstances, specify a list of String-representable Content-Type values using the string-rep element in sipserver.xml. The string-rep element can contain one or more content-type elements to match. If a logged message matches one of the configured content-type elements, Oracle WebLogic Server SIP Container logs the content in String format using UTF-8 encoding, regardless of whether or not a charset parameter is included.

Example 5-5 shows a sample message-debug configuration that logs String content for three additional Content-Type values, in addition to text/* and application/sdp content.

   <message-debug>
     <level>full</level>
     <string-rep>
       <content-type>application/msml+xml</content-type>
       <content-type>application/media_control+xml</content-type>
       <content-type>application/media_control</content-type>
     </string-rep>
   </message-debug>

5.7.5 Enabling Log Rotation and Viewing Log Files

The Oracle WebLogic Server SIP Container logging infrastructure enables you to automatically write to a new log file when the existing log file reaches a specified size. You can also view log contents using the Administration Console or configure additional server-level events that are written to the log.

5.7.6 trace-pattern.dtd Reference

trace-pattern.dtd defines the required contents of the request-pattern.xml and response-pattern.xml, documents, as well as the values for the request-pattern-string and response-pattern-string Servlet init-param variables.

<!--
The different types of conditions supported.
- > 

<!ENTITY % condition "and | or | not |
                      equal | contains | exists | subdomain-of">

<!--
A pattern is a condition: a predicate over the set of SIP requests.
- > 

<!ELEMENT pattern (%condition;)>

<!--
An "and" condition is true if and only if all its constituent conditions
are true.
- > 

<!ELEMENT and (%condition;)+>

<!--
An "or" condition is true if at least one of its constituent conditions
is true.
- > 

<!ELEMENT or (%condition;)+>

<!--
Negates the value of the contained condition.
- > 

<!ELEMENT not (%condition;)>

<!--
True if the value of the variable equals the specified literal value.
- > 

<!ELEMENT equal (var, value)>

<!--
True if the value of the variable contains the specified literal value.
- > 

<!ELEMENT contains (var, value)>

<!--
True if the specified variable exists.
- > 

<!ELEMENT exists (var)>

<!--
- > 

<!ELEMENT subdomain-of (var, value)>

<!--
Specifies a variable. Example:
  <var>request.uri.user</var>
- > 

<!ELEMENT var (#PCDATA)>

<!--
Specifies a literal string value that is used to specify rules.
- > 

<!ELEMENT value (#PCDATA)>

<!--
Specifies whether the "equal" test is case sensitive or not.
- > 

<!ATTLIST equal ignore-case (true|false) "false">

<!--
Specifies whether the "contains" test is case sensitive or not.
- > 

<!ATTLIST contains ignore-case (true|false) "false">

<!--
The ID mechanism is to allow tools to easily make tool-specific
references to the elements of the deployment descriptor. This allows
tools that produce additional deployment information (i.e information
beyond the standard deployment descriptor information) to store the
non-standard information in a separate file, and easily refer from
these tools-specific files to the information in the standard sip-app
deployment descriptor.
- > 

<!ATTLIST pattern id ID #IMPLIED>
<!ATTLIST and id ID #IMPLIED>
<!ATTLIST or id ID #IMPLIED>
<!ATTLIST not id ID #IMPLIED>
<!ATTLIST equal id ID #IMPLIED>
<!ATTLIST contains id ID #IMPLIED>
<!ATTLIST exists id ID #IMPLIED>
<!ATTLIST subdomain-of id ID #IMPLIED>
<!ATTLIST var id ID #IMPLIED>
<!ATTLIST value id ID #IMPLIED>

5.7.7 Adding Tracing Functionality to SIP Servlet Code

Tracing functionality can be added to your own Servlets or to Java code by using the TraceMessageListenerFactory. TraceMessageListenerFactory enables clients to reuse the default trace message listener implementation behaviors by creating an instance and then delegating to it. The factory implementation instance can be found in the servlet context for SIP Servlets by looking up the value of the TraceMessageListenerFactory.TRACE_MESSAGE_LISTENER_FACTORY attribute.

To implement tracing in a Servlet, you use the factory class to create a delegate in the Servlet's init() method as shown in Example 5-7.

public final class TraceMessageListenerImpl extends SipServlet implements MessageListener {
  private MessageListener delegate;

  public void init() throws ServletException {
    ServletContext sc = (ServletContext) getServletContext();
    TraceMessageListenerFactory factory = (TraceMessageListenerFactory) sc.getAttribute(TraceMessageListenerFactory.TRACE_MESSAGE_LISTENER_FACTORY);
    delegate = factory.createTraceMessageListener(getServletConfig());
  }
  public final void onRequest(SipServletRequest req, boolean incoming) {
    delegate.onRequest(req,incoming);
  }
  public final void onResponse(SipServletResponse resp, boolean incoming) {
    delegate.onResponse(resp,incoming);
  }
}

5.7.8 Order of Startup for Listeners and Logging Servlets

If you deploy both listeners and logging servlets, the listener classes are loaded first, followed by the Servlets. Logging Servlets are deployed in order according to the load order specified in their Web Application deployment descriptor.

5.8.1 Modifying JVM Parameters in Server Start Scripts

If you use custom startup scripts to start Oracle WebLogic Server SIP Container engines and replicas, simply edit those scripts to include the recommended JVM options described in the sections that follow.

The Configuration Wizard also installs default startup scripts when you configure a new domain. These scripts are installed in the MIDDLEWARE_HOME/user_projects/domains/domain_name/bin directory by default, and include:

• startWebLogic.cmd, startWebLogic.sh—These scripts start the Administration Server for the domain.

• startManagedWebLogic.cmd, startManagedWebLogic.sh—These scripts start managed engines and replicas in the domain.

If you use the Oracle-installed scripts to start engines and replicas, you can override JVM memory arguments by first setting the USER_MEM_ARGS environment variable in your command shell.

5.8.2 Tuning Garbage Collection with JRockit

JRockit provides several monitoring tools that you can use to analyze the JVM heap at any given moment, including:

• JRockit Runtime Analyzer—provides a view into the runtime behavior of garbage collection and pause times.

• JRockit Stack Dumps—reveals applications' thread activity to help you troubleshoot and/or improve performance.

Use these and other tools in a controlled environment to determine the effects of JVM settings before you use the settings in a production deployment.

The following sections describe suggested starting JVM options for use with the JRockit. If you use JRockit with the deterministic garbage collector (recommended), use the options described in Section 5.8.3, "Using Oracle JRockit Real Time (Deterministic Garbage Collection)".

5.8.3 Using Oracle JRockit Real Time (Deterministic Garbage Collection)

Very short response times are most easily achieved by using JRockit Real Time, which implements a deterministic garbage collector.

Oracle recommends using the following JVM arguments for engine tier servers in replicated cluster configurations:

-Xms1024m -Xmx1024m -XgcPrio:deterministic -XpauseTarget=30ms -XXtlasize:min=8k -XXnosystemgc

For replica servers, increase the available memory:

-Xms3072m -Xmx3072m -XgcPrio:deterministic -XpauseTarget=30ms -XXtlasize:min=8k -XXnosystemgc

These settings fix the heap size and enable the dynamic garbage collector with deterministic garbage collection. -XpauseTarget sets the maximum pause time and -XXtlasize=3k sets the thread-local area size. -XXnosystemgc prevents System.gc() application calls from forcing garbage collection.

5.8.4 Using Oracle JRockit without Deterministic Garbage Collection

When using Oracle's JRockit JVM without deterministic garbage collection (not recommended for production deployments), the best response time performance is obtained by using the generational concurrent garbage collector.

The full list of example startup options for an engine tier server are:

-Xms1024m -Xmx1024m -Xgc:gencon -XXnosystemgc -XXtlasize:min=3k -XXkeeparearatio=0 -Xns:48m 

The full list of example startup options for a replica server are:

-Xms3072m -Xmx3072m -Xgc:gencon -XXnosystemgc -XXtlasize:min=3k -XXkeeparearatio=0 -Xns:48m

5.8.5 Tuning Garbage Collection with Sun JDK

When using Sun's JDK, the goal in tuning garbage collection performance is to reduce the time required to perform a full garbage collection cycle. You should not attempt to tune the JVM to minimize the frequency of full garbage collections, because this generally results in an eventual forced garbage collection cycle that may take up to several full seconds to complete.

The simplest and most reliable way to achieve short garbage collection times over the lifetime of a production server is to use a fixed heap size with the default collector and the parallel young generation collector, restricting the new generation size to at most one third of the overall heap.

The following example JVM settings are recommended for most engine tier servers:

-server -Xmx1024m -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=256 -XX:CMSInitiatingOccupancyFraction=60 -XX:+DisableExplicitGC

For replica servers, use the example settings:

-server -Xmx3072m -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=256 -XX:CMSInitiatingOccupancyFraction=60 -XX:+DisableExplicitGC

The above options have the following effect:

• -XX:+UseTLAB—Uses thread-local object allocation blocks. This improves concurrency by reducing contention on the shared heap lock.

• -XX:+UseParNewGC—Uses a parallel version of the young generation copying collector alongside the concurrent mark-and-sweep collector. This minimizes pauses by using all available CPUs in parallel. The collector is compatible with both the default collector and the Concurrent Mark and Sweep (CMS) collector.

• -Xms, -Xmx—Places boundaries on the heap size to increase the predictability of garbage collection. The heap size is limited in replica servers so that even Full GCs do not trigger SIP retransmissions. -Xms sets the starting size to prevent pauses caused by heap expansion.

• -XX:MaxTenuringThreshold=0—Makes the full NewSize available to every NewGC cycle, and reduces the pause time by not evaluating tenured objects. Technically, this setting promotes all live objects to the older generation, rather than copying them.

• -XX:SurvivorRatio=128—Specifies a high survivor ratio, which goes along with the zero tenuring threshold to ensure that little space is reserved for absent survivors.