Failure of the Oracle WebLogic Server host to respond to a connection request could indicate the following problems:

• Physical problems with the host machine (such as power outages, hardware malfunction, operating system crash, and so on).
• Network problems.
• Other server failures.

Failure of a Oracle WebLogic Server instance to respond could indicate the following problems:

• Oracle WebLogic Server is not running or is unavailable.
• A hung server.
• A database problem.
• An application-specific failure.

Under load, a proxy plug-in may receive CONNECTION_REFUSED errors from a back-end Oracle WebLogic Server instance. For example, the following error is logged in the log file:

weblogic: Trying GET /uri at backend host 'xx.xx.xx.xx/port; got exception 'CONNECTION_REFUSED [os error=xxx, line xxxx of URL.cpp]: apr_socket_connect call failed with error=xxx, host=xx.xx.xx.xx, port=xxxx'

Oracle WebLogic Server might have reached the maximum allowed backlog connections. Follow these tuning tips to reduce CONNECTION_REFUSED errors:

• Increase the AcceptBackLog setting in the configuration of your Oracle WebLogic Server domain.
• Decrease the time wait interval. This setting varies according to the operating system you are using. For example, on Linux, set the net.ipv4.tcp_fin_timeout parameter to a lower value in the /etc/sysctl.conf file.
• Increase the open file descriptor limit on your machine. This limit varies by operating system. Using the limit (.csh) or ulimit (.sh) directives, you can make a script to increase the limit.

If you run only a single Oracle WebLogic Server instance, the proxy plug-in only attempts to connect to the server defined with the WebLogicHost parameter. If the attempt fails, an HTTP 503 error message is returned. The proxy plug-in continues trying to connect to that same Oracle WebLogic Server instance for the maximum number of retries as specified by the ratio of ConnectTimeoutSecs and ConnectRetrySecs.

The WebLogicCluster parameter is required to proxy to a list of back-end servers that are clustered, or to perform load balancing among non-clustered managed server instances.

In the case of proxying to clustered managed servers, when you use the WebLogicCluster parameter to specify a list of Oracle WebLogic Servers, the proxy plug-in uses that list as a starting point for load balancing among the members of the cluster. After the first request is routed to one of these servers, a dynamic server list is returned containing an updated list of servers in the cluster.

The updated list adds any new servers in the cluster and deletes any that have been shut down, or are being suspended, or are no longer part of the cluster or that have failed to respond to requests. This feature can be controlled by using DynamicServerList. For example, to disable this feature, set DynamicServerList to OFF.

DynamicServerList ON is a preferred performance tuning parameter. It is useful, for example, if a member of a cluster is temporarily down for maintenance or if administrators decide they want to add another member, and not need to restart the web server.

When a request contains session information stored in a cookie or in the POST data, or encoded in a URL, the session ID contains a reference to the specific server instance in which the session was originally established (called the primary server). A request containing a cookie attempts to connect to the primary server. If that attempt fails, the proxy plug-in attempts to make a connection to the next available server in the list in a round-robin fashion. That server retrieves the session from the original secondary server and makes itself the new primary server for that same session. See Figure 6-1.

Figure 6-1 Connection Failover

Description of "Figure 6-1 Connection Failover"

In this figure, the Maximum number of retries allowed in the red loop is equal to ConnectTimeoutSecs/ConnectRetrySecs.

In some configurations that use combinations of firewalls and load-directors, any one of the servers (firewall or load-directors) can accept the request and return a successful connection while the primary instance of Oracle WebLogic Server is unavailable. After attempting to direct the request to the primary instance of Oracle WebLogic Server (which is unavailable), the request is returned to the proxy plug-in as "connection reset."

Requests running through combinations of firewalls (with or without load-directors) are handled by Oracle WebLogic Server. In other words, responses of connection reset fail over to a secondary instance of Oracle WebLogic Server. Because responses of connection reset fail over in these configurations, servlets must be idempotent. Otherwise duplicate processing of transactions may result.