Managing Server Startup and Shutdown

Avoiding and Recovering From Server Failure

A variety of events can lead to the failure of a server instance. Often one failure condition leads to another. Loss of power, hardware malfunction, operating system crashes, network partitions, and unexpected application behavior can all contribute to the failure of a server instance.

For high availability requirements, implement a clustered architecture to minimize the impact of failure events. (For information about failover in a WebLogic Server cluster, see Failover and Replication in a Cluster in Using WebLogic Server Clusters.) However, even in a clustered environment, server instances may fail periodically, and it is important to be prepared for the recovery process.

The following sections provide information and procedures for recovering failed server instances:

Failure Prevention and Recovery Features

WebLogic Server offers several features that facilitate recovery from and protection against server failure.

Overload Protection

WebLogic Server 9.0 detects increases in system load that can affect application performance and stability, and allows administrators to configure failure prevention actions that occur automatically at predefined load thresholds.

Overload protection helps you avoid failures that result from unanticipated levels of application traffic or resource utilization.

WebLogic Server attempts to avoid failure when certain conditions occur:

Workload manager capacity is exceeded
HTTP session count increases to a predefined threshold value
Impending out of memory conditions

Failover for Clustered Services

You can increase the reliability and availability of your applications by hosting them on a WebLogic Server cluster. Clusterable services, such as EJBs and Web applications, can be deployed uniformly—on each Managed Server—in a cluster, so that if the server instance upon which a service is deployed fails, the service can fail over to another server in the cluster, without interruption in service or loss of state.

For more information, see Failover and Replication in a Cluster in Using WebLogic Server Clusters.

Automatic Restart for Failed Server Instances

WebLogic Server self-health monitoring improves the reliability and availability of server instances in a domain. Selected subsystems within each WebLogic Server instance monitor their health status based on criteria specific to the subsystem. For example, the JMS subsystem monitors the condition of the JMS thread pool while the core server subsystem monitors default and user-defined execute queue statistics. If an individual subsystem determines that it can no longer operate in a consistent and reliable manner, it registers its health state as "failed" with the host server.

Each WebLogic Server instance, in turn, checks the health state of its registered subsystems to determine its overall viability. If one or more of its critical subsystems have reached the FAILED state, the server instance marks its own health state FAILED to indicate that it cannot reliably host an application.

Using Node Manager, server self-health monitoring enables you to automatically reboot servers that have failed. This improves the overall reliability of a domain, and requires no direct intervention from an administrator.

For more information, see Using Node Manager to Control Servers.

Server-Level Migration

WebLogic Server 9.0 provides the capability to migrate clustered server instances. A clustered server that is configured to be migratable can be moved in its entirety from one machine to another, at the command of an administrator, or automatically, in the event of failure. The migration process makes all of the services running on the server instance available on a different machine, but not the state information for the singleton services that were running at the time of failure. For more information, see Server Migration in Using WebLogic Server Clusters.

Service-Level Migration

WebLogic Server supports migration of a individual singleton service as well as the server-level migration capability described in the previous section. Singleton services are services that run in a cluster but must run on only a single instance at any given time, such as JMS and the JTA transaction recovery system.

An administrator can migrate a JMS server or the JTS transaction recovery from one server instance to another in a cluster, either in response to a server failure or as part of regularly-scheduled maintenance. This capability improves the availability of pinned services in a cluster, because those services can be quickly restarted on a redundant server should the host server fail.

Managed Server Independence Mode

Managed Servers maintain a local copy of the domain configuration. When a Managed Server starts, it contacts its Administration Server to retrieve any changes to the domain configuration that were made since the Managed Server was last shut down. If a Managed Server cannot connect to the Administration Server during startup, it can use its locally cached configuration information—this is the configuration that was current at the time of the Managed Server's most recent shutdown. A Managed Server that starts up without contacting its Administration Server to check for configuration updates is running in Managed Server Independence (MSI) mode. By default, MSI mode is enabled. For information about disabling MSI mode, see Disabling Managed Server Independence in Administration Console Online Help.

Directory and File Backups for Failure Recovery

Recovery from the failure of a server instance requires access to the domain's configuration and security data. This section describes file backups that WebLogic Server performs automatically, and recommended backup procedures that an administrator should perform.

Recovery from the failure of a server instance requires access to the domain's configuration and security data. The WebLogic Security service stores its configuration data in the config.xml file, and also in an LDAP repository and other files.

For more information, see "Domain Configuration Files" and in Understanding Domain Configuration.

Back Up Domain Configuration Directory

By default, an Administration Server stores a domain's configuration data in the domain_name\config directory, where domain_name is the root directory of the domain.

Back up the config directory to a secure location in case a failure of the Administration Server renders the original copy unavailable. If an Administration Server fails, you can copy the backup version to a different machine and restart the Administration Server on the new machine.

Each time a Managed Server starts up, it contacts the Administration Server and if there are changes in to the domain configuration, the Managed Server updates its local copy of the domain config directory.

During operation, if changes are made to the domain configuration, the Administration Server notifies the Managed Servers which update their local /config directory. So, each Managed Server always has an current copy of its configuration data cached locally.

Back Up LDAP Repository

The default Authentication, Authorization, Role Mapper, and Credential Mapper providers that are installed with WebLogic Server store their data in an LDAP server. Each WebLogic Server contains an embedded LDAP server. The Administration Server contains the master LDAP server which is replicated on all Managed Servers. If any of your security realms use these installed providers, you should maintain an up-to-date backup of the following directory tree:

domain_name\servers\adminServer\data\ldap

where domain_name is the domain's root directory and adminServer is the directory in which the Administration Server stores runtime and security data.

Each WebLogic Server has an LDAP directory, but you only need to back up the LDAP data on the Administration Server—the master LDAP server replicates the LDAP data from each Managed Server when updates to security data are made. WebLogic security providers cannot modify security data while the domain's Administration Server is unavailable. The LDAP repositories on Managed Servers are replicas and cannot be modified.

The ldap\ldapfiles subdirectory contains the data files for the LDAP server. The files in this directory contain user, group, group membership, policies, and role information. Other subdirectories under the ldap directory contain LDAP server message logs and data about replicated LDAP servers.

Do not update the configuration of a security provider while a backup of LDAP data is in progress. If a change is made—for instance, if an administrator adds a user—while you are backing up the ldap directory tree, the backups in the ldapfiles subdirectory could become inconsistent. If this does occur, consistent, but potentially out-of-date, LDAP backups are available, because once a day, a server suspends write operations and creates its own backup of the LDAP data. It archives this backup in a ZIP file below the ldap\backup directory and then resumes write operations. This backup is guaranteed to be consistent, but it might not contain the latest security data.

For information about configuring the LDAP backup, see Configuring Backups for the Embedded LDAP Server in Administration Console Online Help.

Back Up SerializedSystemIni.dat and Security Certificates

Each server instance creates a file named SerializedSystemIni.dat and locates it in the /security directory. This file contains encrypted security data that must be present to boot the server. You must back up this file.

If you configured a server to use SSL, you must also back up the security certificates and keys. The location of these files is user-configurable.

WebLogic Server Exit Codes and Restarting After Failure

When a server instance stops, it issues an exit code. The value of the exit code provides information about the conditions under which the server process ended. When a server instance under Node Manager control exits, Node Manager uses the exit code to determine whether or not to restart the server instance. The server exit code can be used by other high-availability agents or scripts to determine what, if any action, to take after a server instance exits. Server exit codes are defined in Table 4-1.

Table 4-1 WebLogic Server Exit Codes

Exit Code Value	Meaning	Restart Recommendation
Less than 0	A negative value indicates that the server instance failed during a state transition, and did not terminate in a stable condition. Example: If a Start in Standby command is issued for a server instance whose configuration is invalid, the server instance fails in the transitional `STARTING` state, and does not achieve the `STANDBY` state.	Do not attempt to restart the server. Diagnose the problem that caused the server process to exit.
0	Indicates that the server process terminated normally, as a result of a shutdown command, either graceful or forced.	None.
Greater than 0	A positive value indicates that the server instance stopped itself after determining that one or more of its subsystems were unstable. Example: A server instance detects an out of memory condition or stuck threads, and shuts itself down.	The server instance can be restarted.

Exit Code Value

Meaning

Restart Recommendation

Less than 0

A negative value indicates that the server instance failed during a state transition, and did not terminate in a stable condition.

Example: If a Start in Standby command is issued for a server instance whose configuration is invalid, the server instance fails in the transitional STARTING state, and does not achieve the STANDBY state.

Do not attempt to restart the server. Diagnose the problem that caused the server process to exit.

Indicates that the server process terminated normally, as a result of a shutdown command, either graceful or forced.

None.

Greater than 0

A positive value indicates that the server instance stopped itself after determining that one or more of its subsystems were unstable.

Example: A server instance detects an out of memory condition or stuck threads, and shuts itself down.

The server instance can be restarted.

Restarting a Failed Administration Server

The following sections describe how to start an Administration Server after a failure.

Note: You can use Node Manager to automatically restart a failed Administration Server. For more information see Using Node Manager to Control Servers.

Restarting an Administration Server

See Starting and Stopping Servers.

Restarting an Administration Server on the Same Machine

Table 4-2 Starting Up

Listen Address Definition	Admin Server Restart Scenario
	Same Machine	Different Machine
Not defined	1. Start the Administration Server. Running MSs will reconnect automatically at the next `ReconnectIntervalSecs` To start an MS that was not running when AS failed, no change in command is required.	1. Install WLS. 2. Move data. 3. Start the Admin Server. Running MSs will learn the new AS address when they are contacted by the AS after it has been started. To start an MS that was not running when AS failed, supply the new AS Listen Address on command line.
DNS name or IP address of the host	1. Start the Admin Server. Running MSs will reconnect automatically at the next `ReconnectIntervalSecs` To start an MS that was not running when AS failed, no change in command is required	1. Install WLS. 2. Move data. 3. Move IP address. Running MSs will reconnect automatically at next `ReconnectIntervalSecs` To start an MS that was not running when AS failed, no change in command is required 4. If you do not move the IP address Running MSs will learn the new AS address when they are contacted by the AS after it has been started. To start an MS that was not running when AS failed, you must supply the new Listen Address on the command line.
DNS name mapped to multiple hosts	1. Start the Admin Server. Running MSs will reconnect automatically at the next `ReconnectIntervalSecs` To start an MS that was not running when AS failed, no change in command is required	1. Install WLS. 2. Move data. Running MSs will reconnect automatically at next `ReconnectIntervalSecs` To start an MS that was not running when AS failed, no change in command is required.

Restarting an Administration Server on Another Machine

If a machine crash prevents you from restarting the Administration Server on the same machine, you can recover management of the running Managed Servers as follows:

Install the WebLogic Server software on the new administration machine (if this has not already been done).

Make your application files available to the new Administration Server by copying them from backups or by using a shared disk. Your application files should be available in the same relative location on the new file system as on the file system of the original Administration Server.

Make your configuration and security data available to the new administration machine by copying them from backups or by using a shared disk. For more information, refer to Directory and File Backups for Failure Recovery.

Restart the Administration Server on the new machine.

Managed Servers and Re-started Administration Server

If an Administration Server stops running while the Managed Servers in the domain continue to run, each Managed Server periodically attempts to reconnect to the Administration Server, at the interval specified by the ServerMBean attribute AdminReconnectIntervalSecs. By default, AdminReconnectIntervalSecs is ten seconds.

When the Administration Server starts, it communicates with the Managed Servers and informs them that the Administration Server is now running on a different IP address.

Restarting a Failed Managed Server

The following sections describe how to start Managed Servers after failure. For recovery considerations related to transactions and JMS, see Additional Failure Topics.

Starting a Managed Server When the Administration Server Is Accessible

If the Administration Server is reachable by Managed Server that failed, you can:

Restart it manually or automatically using Node Manager—You must configure Node Manager and the Managed Server to support this behavior. For details, see Start, Shut Down, Suspend, and Restart Managed Servers.
Start it manually with a command or script—For instructions, see Starting and Stopping Servers.

Starting a Managed Server When the Administration Server Is Not Accessible

If a Managed Server cannot connect to the Administration Server during startup, it can retrieve its configuration by reading its locally cached configuration data from the config directory. A Managed Server that starts in this way is running in Managed Server Independence (MSI) mode.

Understanding Managed Server Independence Mode

When a Managed Server starts, it tries to contact the Administration Server to retrieve its configuration information. If a Managed Server cannot connect to the Administration Server during startup, it can retrieve its configuration by reading configuration and security files directly. A Managed Server that starts in this way is running in Managed Server Independence (MSI) mode. By default, MSI mode is enabled. For information about disabling MSI mode, see Disabling Managed Server Independence in Administration Console Online Help.

In Managed Server Independence mode, a Managed Server:

looks in its local config directory for config.xml—a replica of the domain's config.xml.
looks in its security directory for SerializedSystemIni.dat and for boot.properties, which contains an encrypted version of your username and password. For more information, see Boot Identity Files.

If config.xml and SerializedSystemIni.dat are not in these locations in the server's domain directory, you can copy them from the Administration Server's domain directory.

MSI Mode and Node Manager

You cannot use Node Manager to start a server instance in MSI mode, only to restart it. For a routine startup, Node Manager requires access to the Administration Server. If the Administration Server is unavailable, you must log onto a Managed Server's host machine to start the Managed Server.

MSI Mode and the Security Realm

A Managed Server must have access to a security realm to complete its startup process.

If you use the security realm that WebLogic Server installs, then the Administration Server maintains an LDAP server to store the domain's security data. All Managed Servers replicate this LDAP server. If the Administration Server fails, Managed Servers running in MSI mode use the replicated LDAP server for security services.

If you use a third party security provider, then the Managed Server must be able to access the security data before it can complete its startup process.

MSI Mode and SSL

If you set up SSL for your servers, each server requires its own set of certificate files, key files, and other SSL-related files. Managed Servers do not retrieve SSL-related files from the Administration Server though the domain's configuration file does store the pathnames to those files for each server. Starting in MSI Mode does not require you to copy or move the SSL-related files unless they are located on a machine that is inaccessible.

MSI Mode and Deployment

A Managed Server that starts in MSI mode deploys its applications from its staging directory: serverroot\stage\appName.

MSI Mode and the Domain Log File

Each WebLogic Server instance writes log messages to its local log file and a domain-wide log file. The domain log file provides a central location from which to view messages from all servers in a domain.

Usually, a Managed Server forwards messages to the Administration Server, and the Administration Server writes the messages to the domain log file. However, when a Managed Server runs in MSI mode, it continues to write messages to its local server log file but does not forward messages to the domain log file.

For more information, see "How a Server Instance Forwards Messages to the Domain Log" in Configuring Log Files and Filtering Log Messages.

MSI Mode and Managed Server Configuration Changes

If you start a Managed Server in MSI mode, you cannot change its configuration until it restores communication with the Administration Server.

Starting a Managed Server in MSI Mode

Note: If the Managed Server that failed was a clustered Managed Server that was the active server for a migratable service at the time of failure, perform the steps described in Migrating When the Currently Active Host is Unavailable in Using WebLogic Server Clusters. Do not start the Managed Server in MSI mode.

To start up a Managed Server in MSI mode:

Ensure that the Managed Server's root directory contains the config subdirectory.

If the config directory does not exist, copy it from the Administration Server's root directory or from a backup to the Managed Server's root directory.

Note: Alternatively, you can use the -Dweblogic.RootDirectory=path startup option to specify a root directory that already contains these files.

Start the Managed Server at the command line or using a script.

The Managed Server will run in MSI mode until it is contacted by its Administration Server. For information about restarting the Administration Server in this scenario, see Restarting a Failed Administration Server.

Additional Failure Topics

For information related to recovering JMS data from a failed server instance, see Configuring Clustered WebLogic JMS Resources in Programming WebLogic JMS.

For information about transaction recovery after failure, see "Transaction Recovery After a Server Fails" in Programming WebLogic JTA.