Configuring and Managing WebLogic SIP Server

Configuring SNMP

The following sections describe how to configure and manage SNMP services with WebLogic SIP Server 2.1:

Overview of WebLogic SIP Server SNMP

WebLogic SIP Server includes a dedicated SNMP MIB to monitor activity on engine tier and data tier server instances. The WebLogic SIP Server MIB adds several traps and attributes in addition to those provided by the WebLogic Server 8.1 MIB. Engine and data tier traps are generated directly by the Managed Server instances that make up each tier.

The Administration Server may also generate traps on behalf of the WebLogic SIP Server domain as a whole, for example when a server in a cluster fails. WebLogic SIP Server MIB entries can also be monitored directly from the Administration Server of a domain.

Note: WebLogic SIP Server MIB objects are read-only. You cannot modify a WebLogic SIP Server configuration using SNMP.

Browsing the MIB

You can use either of the following methods to browse the contents of the WebLogic SIP Server MIB:

Use a MIB browser. WebLogic SIP Server does not provide a MIB browser, but most vendors of SNMP utilities do. The MIB is located in a file named WLSS_HOME/telco/lib/BEA-WLSS-MIB.asn1.
Use a Web browser to view the WebLogic SIP Server SNMP MIB Reference on the BEA e-docs Web site.

Because the MIB Reference uses Javascript and DHTML to provide browsing capabilities that are similar to a MIB browser, you must use one of the following Web browsers:

Firefox
Internet Explorer, version 5 or higher
Mozilla
Netscape Navigator, version 6 or higher
Opera 7 or higher

Configuring SNMP

To enable SNMP monitoring for the entire WebLogic SIP Server domain, follow these steps:

In the left pane, select the Services->SNMP node.

Select the Enabled check box to enable SNMP.

Note: WebLogic SIP Server instances ignore the SNMP port number specified on this page. See SNMP Port Binding for WebLogic SIP Server.

Click Apply to apply your changes.

SNMP Port Binding for WebLogic SIP Server

If you run multiple Managed Server instances on the same machine, each server instance would normally attempt to bind to the same configured SNMP port (for example, port 161). WebLogic SIP Server instances automatically manage SNMP port conflicts by automatically attempting to bind to port 1610, and incrementing the port number as needed if the current port is unavailable. This helps to avoid a SNMP startup failure when multiple WebLogic SIP Server instances are deployed on the same server hardware.

You can also manually override the starting port number that WebLogic SIP Server attempts to bind to by supplying the -DWLSS.SNMPPort=port_number startup argument.

Warning: If you specify the -DWLSS.SNMPPort option, ensure that the starting port number and subsequent numbers are unused on your system. The default starting port of 1610 was selected because no services are commonly bound to the 1610 port range.

Understanding and Responding to SNMP Traps

The following sections describe the WebLogic SIP Server SNMP traps in more detail. Recovery procedures for responding to individual traps are also included where applicable.

Files for Troubleshooting

The following WebLogic SIP Server log and configuration files are frequently helpful for troubleshooting problems, and may be required by your technical support contact:

$DOMAIN_DIR/config.xml
$DOMAIN_DIR/sipserver/config/sipserver.xml
$DOMAIN_DIR/*.log (domain log file)
$DOMAIN_DIR/servername/*.log (server and access logs)
sip.xml (in the /WEB-INF subdirectory of the application)
web.xml (in the /WEB-INF subdirectory of the application)

General information that can help the technical support team includes:

The specific versions of:

WebLogic SIP Server
Java SDK
Operating System

Thread dumps for hung WebLogic SIP Server processes
Network analyzer logs

Trap Descriptions

The following sections provide detailed information about the following WebLogic SIP Server SNMP traps:

sipAppDeployed

Description

WebLogic SIP Server generates this alarm when a SIP Servlet is deployed to the container.

Recovery Procedure

This trap is generated during normal deployment operations and does not indicate an exception.

sipAppUndeployed

Description

WebLogic SIP Server generates this alarm when a SIP application shuts down, or if a SIP application is undeployed. This generally occurs when WebLogic SIP Server is shutdown while active requests still exist.

Recovery Procedure

During normal shutdown procedures this alarm should be filtered out and should not reach operations. If the alarm occurs during the course of normal operations, it indicates that someone has shutdown the application or server unexpectedly, or there is a problem with the application. Notify Tier 4 support immediately.

sipAppFailedToDeploy

Description

WebLogic SIP Server generates this trap when an application deploys successfully as a Web Application but fails to deploy as a SIP application.

Recovery Procedure

The typical failure is caused by an invalid sip.xml configuration file and should occur only during software installation or upgrade procedures. When it occurs, undeploy the application, validate the sip.xml file, and retry the deployment.

Note: This alarm should never occur during normal operations. If it does, contact Tier 4 support immediately.

overloadControlActivated, overloadControlDeactivated

Description

Weblogic SIP Server uses a configurable throttling mechanism that helps you control the number of new SIP requests that are processed. After a configured overload condition is observed, WebLogic SIP Server destroys new SIP requests by responding with "503 Service Unavailable" to the caller. The server continues to destroy new requests until the overload condition is resolved according to a configured threshold control value. This alarm is generated when the throttling mechanism is activated. The throttling behavior should eventually return the server to a non-overloaded state, and further action may be unnecessary. See Overload in Configuring and Managing WebLogic SIP Server.

Recovery Procedure

1. Check other servers to see if they are nearly overloaded.

2. Check to see if the load balancer is correctly balancing load across the application servers, or if it is overloading one or more servers. If additional servers are nearly overloaded, Notify Tier 4 support immediately.

3. If the issue is limited to one server, notify Tier 4 support within one hour.

Additional Overload FAQs

Question: How can I monitor load using the Administration Server? How can I tell when I'm near a threshold?

Answer: If you set the queue length as an incoming call overload control, you can monitor the length of the queue using the Administration Console. If you specify a session rate control, you cannot monitor the session rate using the Administration Console. (The Administration Console only displays the current number of SIP sessions, not the rate of new sessions generated.)

licenseLimitExceeded

Description

WebLogic SIP Server generates this trap when it detects a license violation. This trap can occur if server usage or access exceeds the limitations specified in the license file, or if the file is accidentally modified. Never modify, move, or delete the license file during normal operations.

License violations may cause one or more of the following behaviors if the license file has been modified or corrupted:

If the license signature in the license file was changed, WebLogic SIP Server may detect the anomaly and shut down. Furthermore, WebLogic SIP Server will not start up if the license signature is modified or corrupted.
If the IP address in the license file is incorrect, WebLogic SIP Server will not start up.
If the license expiration is reached WebLogic SIP Server will not start up. If the server is already running, it will shut down automatically after the expiration is reached.

Note: Permanent licenses have no expiration. If you purchased a permanent license but your license expires, your server may be using an evaluation license instead of your purchased license.

If usage reaches the maximum values set in the license file (max-sessions, max-registers or max-users) WebLogic SIP Server continues to run but rejects requests that exceed the defined limits. The following behaviors may be observed when usage limits are reached:

If the number of sessions has reached max-sessions and there is a request to create a new session:

WebLogic SIP Server generates a licenseLimitExceeded exception and rejects the request.
The application must handle the exception and notify the source of the request.
If the application passes the exception to the SIP container as is, WebLogic SIP Server returns a "503 Out Of Licensed Resources" response.

If there is an attempt to register a user beyond the maximum number specified in max-users:

The user management component generates an IllegalLicenseException.
The application must handle the exception and notify the source of the request.
If the application passes the exception to the SIP container as is, WebLogic SIP Server returns a "503 Out Of Licensed Resources" response.
Removing an existing user enables a new user to be registered.

If the number of connected terminals has reached max-registers:

The registrar servlet returns "503 Service Unavailable" to the new REGISTER request.
Registered terminals can still be refreshing or removed (UNREGISTERed).
UNREGISTERing a terminal enables a new REGISTER to succeed.

The license file is an XML document located at $BEA_HOME/license.bea. The following sample shows a portion of an evaluation license:

    <license-group format="1.0" product="WebLogic SIP Server" release="2.1">

        <license

            component="SIP Servlet Engine"

            expiration="never"

            ip="any"

            licensee="BEA Evaluation Customer"

            msgspersec="100"

            serial="616351266349-1813874379535"

            type="SDK"

            signature="MC0CFQDeWBkXTSZ5bO1qy0D/AfukgzqhDwIURsL8bkpTwlypiTSBq+d1b

dyzbRM="

/>

    </license-group>

</license-group>

Recovery Procedure

Check the license file to insure that it has not been accidentally removed, changed or corrupted. Note that WebLogic SIP Server checks the license every four hours, so repairs may not be registered immediately.

Check the expiration date in the license, and confirm that an EVAL license was not accidentally installed over the permanent license.

Notify Tier 4 Support of the condition, and send them a copy of the license file.

Additional License FAQs

Question: What IP should be used for licensing purposes?

Question: Each box has multiple IP addresses. Which IP should be assigned to the license?

Answer: Use the IP address that is returned with get local host command.

Question: I've upgraded my hardware system or need to move WebLogic SIP Server to a new machine. How do I modify the license file to use a new IP address?

Answer: Contact BEA Support with the updated IP address. BEA will generate a new license for you. You can then replace the license file with the updated file immediately without stopping WebLogic SIP Server.

serverStopped

Description

This trap indicates that the WebLogic Server instance is now down. This trap applies to both engine tier and data tier server instances. If this trap is received spontaneously and not as a result of a controlled shutdown, follow the steps below.

Recovery Procedure

Use the following command to identify the hung process:

ps -ef | grep java

There should be only one PID for each WebLogic Server instance running on the machine.

After identifying the affected PID, use the following command to kill the process:

kill -3 [pid]

This command generates the actual thread dump. If the process is not immediately killed, repeat the command several times, spaced 5-10 seconds apart, to help diagnose potential deadlock problems, until the process is killed.

Attempt to restart WebLogic SIP Server immediately. See Restarting Failed Server Instances in the WebLogic Server 8.1 documentation.

Make a backup copy of all SIP logs on the affected server to aid in troubleshooting. The location of the logs varies based on the server configuration.

Copy each log to assist Tier 4 support with troubleshooting the problem.

Note: WebLogic SIP Server logs are truncated according to your system configuration. Make backup logs immediately to avoid losing critical troubleshooting information.

Notify Tier 4 support and include the log files with the trouble ticket.

Monitor the server closely over next 24 hours. If the source of the problem cannot be identified in the log files, there may be a hardware or network issue that will reappear over time.

Additional Shutdown FAQs

Question: If the server shuts down, are all SNMP traps for the server lost?

Answer: The Administration Console generates SNMP messages for managed WebLogic Server instances only until the ServerShutDown message is received. Afterwards, no additional messages are generated.

dataTierServerStopped

Description

WebLogic SIP Server generates this alarm when an unrecoverable error occurs in a WebLogic Server instance that is part of the data tier.

Recovery Procedure

See the Recovery Procedure for serverStopped.

replicaAddedToPartition

Description

WebLogic SIP Server generates this alarm when a server instance is added to a partition in the data tier.

Recovery Procedure

This trap is generated during normal startup procedures when data tier servers are booted.

replicaRemovedFromPartition

Description

This trap is generated when a WebLogic SIP Server instance is removed from the data tier, either as a result of a normal shutdown operation or because of a failure.

Recovery Procedure

If this trap is generated as a result of a server instance failure, additional traps will be generated to indicate the exception. See the recovery procedures for traps generated in addition to replicaRemovedFromPartition.

connectionLostToPeer

Description

This trap is generated when an engine tier server instance loses its connection to a replica in the data tier. It may indicate a network connection problem between the engine and data tiers, or may be generated with additional traps if a data tier server fails.

Recovery Procedure

If this trap occurs in isolation from other traps indicating a server failure, it generally indicates a network failure. Verify or repair the network connection between the affected engine tier server and the data tier server.

If the trap is accompanied by additional traps indicating a data tier server failure (for example, dataTierServerStopped), follow the recovery procedures for the associated traps.

connectionReestablishedToPeer

Description

This trap is generated when an engine tier server reconnects to a data tier server after a prior failure (after a connectionLostToPeer trap was generated). Repeated instances of this trap may indicate an intermittent network failure between the engine and data tiers.

Recovery Procedure

See connectionLostToPeer.