18 Using Service Migration in an Enterprise Deployment

The Oracle WebLogic Server migration framework supports Whole Server Migration and Service Migration. Whole Server Migration requires more resources and a full start of a managed server, so it involves a higher failover latency than Service Migration. The products included in this EDG support Service Migration. Hence, Service Migration is recommended and this guide explains how to use Service Migration in an Oracle Fusion Middleware enterprise topology. Whole Server migration is out of the scope of this guide.

About Automatic Service Migration in an Enterprise Deployment

Oracle WebLogic Server provides a migration framework that is an integral part of any highly available environment. The following sections provide more information about how this framework can be used effectively in an enterprise deployment.

Understanding the Difference between Whole Server and Service Migration

The Oracle WebLogic Server migration framework supports two distinct types of automatic migration:

  • Whole Server Migration, where the Managed Server instance is migrated to a different physical system upon failure.

    Whole server migration provides for the automatic restart of a server instance, with all its services, on a different physical machine. When a failure occurs in a server that is part of a cluster which is configured with server migration, the server is restarted on any of the other machines that host members of the cluster.

    For this to happen, the servers must use a floating IP as listen address and the required resources (transactions logs and JMS persistent stores) must be available on the candidate machines.

    See Whole Server Migration in Administering Clusters for Oracle WebLogic Server.

  • Service Migration, where specific services are moved to a different Managed Server within the cluster.

    To understand service migration, it's important to understand pinned services.

    In a WebLogic Server cluster, most subsystem services are hosted homogeneously on all server instances in the cluster, enabling transparent failover from one server to another. In contrast, pinned services, such as JMS-related services, the JTA Transaction Recovery Service, and user-defined singleton services, are hosted on individual server instances within a cluster—for these services, the WebLogic Server migration framework supports failure recovery with service migration, as opposed to failover.

    See Understanding the Service Migration Framework in Administering Clusters for Oracle WebLogic Server.

Implications of Service Migration in an Enterprise Deployment

Using Automatic Service Migration (ASM) in an Enterprise Deployment has implications in the infrastructure and configuration requirements.

The implications are:

  • The resources used by servers must be accessible to both the original and failover system

    In its initial status, resources are accessed by the original server or service. When a server or service is failed over/restarted in another system, the same resources (such as external resources, databases, and stores) must be available in the failover system. Otherwise, the service cannot resume the same operations. It is for this reason, that both whole server and service migration require that all members of a WebLogic cluster have access to the same transaction and JMS persistent stores.

    Oracle allows you to use JDBC stores, which leverage the consistency, data protection, and high availability features of an oracle database and makes resources available for all the servers in the cluster. Alternatively, you can use shared storage. When you configure persistent stores properly in the database or in shared storage, you must ensure that if a failover occurs (whole server migration or service migration), the failover system is able to access the same stores without any manual intervention.

  • Leasing Datasource

    Service migration requires the configuration of a leasing datasource that is used by servers to store alive timestamps. These timestamps are used to determine the health of a server or service, and are key to the correct behavior of server and service migration (they are used to marks servers or services as failed and trigger failover).

    Note:

    Oracle does not recommend that you use consensus leasing for HA purposes.

Understanding Which Products and Components Require Whole Server Migration and Service Migration

The following table summarizes the list of FMW products and components that benefit from use of a migration capability and indicates the best-practice recommendation for this release. Components listed as migratable can use either Whole Server or Automatic Service Migration.

Note that the table lists the recommended best practice. It does not preclude you from using Whole Server or Automatic Server Migration for those components that support it.

Component Whole Server Migration (WSM) Automatic Service Migration (ASM)

Oracle Web Services Manager (OWSM)

NO

NO

Oracle WebCenter Portal

NO

NO

Oracle WebCenter Portal Portlets and Pagelet Producers

NO

NO

Oracle WebCenter Content

YES

YES (Recommended)

Oracle WebCenter Inbound Refinery

NO

NO

Oracle SOA Suite

YES

YES (Recommended)

Oracle Enterprise Scheduler

NO

NO

Creating a GridLink Data Source for Leasing

Automatic Service Migration requires a data source for the leasing table, which is a table that resides in a tablespace, and schema created automatically, as part of the Oracle WebLogic Server schemas by the Repository Creation Utility (RCU).

Note:

To accomplish data source consolidation and connection usage reduction, you can reuse the WLSSchemaDatasource as is for database leasing. This data source is already configured with the FMW1412_WLS_RUNTIME schema, where the leasing table is stored.

For an enterprise deployment, you should create a GridLink data source:

  1. Log in to the WebLogic Remote Console.
  2. Navigate to the Edit Tree.
  3. In the structure tree, expand Services and select Data Sources.
  4. On the Summary of Data Sources page, click New and select GridLink Data Source. Enter the following:

    Table 18-1 GridLink Data Source Properties

    Properties Description
    Name Enter a logical name for the data source in the Name field. For example, Leasing.
    JNDI Names Enter a name for JNDI. For example, jdbc/leasing.
    Targets Select the cluster that you are configuring for Automatic Service Migration and move to "Chosen".
    Data Source Type Select GridLink Data Source.
    Database Driver Select Oracle's Driver (Thin) for GridLink Connections Versions: Any.
    Global Transaction Protocol Select None.
    Listeners Enter the SCAN address and port for the RAC database, separated by a colon. For example, db-scan.example.com:1521.
    Service Name Enter the service name of the database with lowercase characters. For a GridLink data source, you must enter the Oracle RAC service name. For example, soaedg.example.com.
    Database username

    Enter the user name of the WLS Runtime schema. For example, FMW1412_WLS_RUNTIME. In this example, FMW1412 is the prefix you used when you created the schemas as you prepared to configure the domain.

    Note:

    The leasing table is created automatically when you create the WLS schemas with the Repository Creation Utility (RCU).
    Password Enter the password you used when you created the WLS schema in RCU.
    Protocol Leave the default value (TCP).
    Fan Enabled You must check this option.
    ONS Nodes Leave it empty. ONS node list is automatically retrieved when the database is 12.2 or higher version.
    ONS Wallet and password You can leave this field empty.
    Test Configuration You must enable this option.
  5. Click Create.
  6. Commit changes in the shopping cart.

Configuring Automatic Service Migration in an Enterprise Deployment

You may need to configure automatic service migration for specific services in an enterprise deployment.

Note:

The Automatic Service Migration feature is part of Oracle Integration Continuous Availability. For more details on Oracle SOA Suite for Middleware Options, see Oracle Fusion Middleware Licensing Information.

Setting the Leasing Mechanism and Data Source for an Enterprise Deployment Cluster

Before you can configure automatic service migration, you must verify the leasing mechanism and data source that is used by the automatic service migration feature. You must configure the leasing mechanism and datasource for both static and dynamic clusters.

Note:

To accomplish data source consolidation and connection usage reduction, you can reuse the WLSSchemaDatasource datasource as is for database leasing. This datasource is already configured with the FMW1412_WLS_RUNTIME schema, where the leasing table is stored.

The following procedure assumes that you have configured the Leasing data source either by reusing the WLSSChemaDatasource or a custom datasource that you created as described in Creating a GridLink Data Source for Leasing.

  1. Log in to the WebLogic Remote Console.
  2. Navigate to the Edit Tree.
  3. In the structure tree, expand Environment > Clusters.

    The Summary of Clusters page appears.

  4. Click the cluster for which you want to configure migration.
  5. Click the Migration tab.
  6. Verify that Database is selected in the Migration Basis drop-down menu.
  7. In the Data Source for Automatic Migration drop-down menu, select the Leasing data source that you created in Creating a GridLink Data Source for Leasing. Select the WLSRuntimeSchemaDataSource for data source consolidation.
  8. Click Save.
  9. Commit changes in the shopping cart.
  10. Restart the managed servers for the changes to be effective. If you are configuring other aspects of ASM in the same configuration change session, you can use a final unique restart to reduce downtime.

After you complete the database leasing configuration, continue with the configuration of the service migration:

Configuring Automatic Service Migration

After you have configured the leasing for the cluster as described in Setting the Leasing Mechanism and Data Source for an Enterprise Deployment Cluster, you can configure automatic service migration for specific services in an enterprise deployment. The following sections explain how to configure and validate Automatic Service Migration for static clusters.

Changing the Migration Settings for the Managed Servers in the Cluster

After you set the leasing mechanism and data source for the cluster, you can then enable automatic JTA migration for the Managed Servers that you want to configure for service migration. Note that this topic applies only if you are deploying JTA services as part of your enterprise deployment.

To change the migration settings for the Managed Servers in each cluster:
  1. Log in to the WebLogic Remote Console.
  2. Navigate to the Edit Tree.
  3. In the structure tree, expand the Environment node and click Servers.
    The Summary of Servers page appears.
  4. Expand the name of the server you want to modify.
  5. Navigate to JTA Migratable Target.
  6. In the the JTA Migration Policy drop-down menu, select Failure Recovery.

    In the JTA Candidate Servers section of the page, leave the Chosen list box empty. If you do not select any servers from the Available list box, all the available servers in the cluster become candidates for service migration.

  7. Click Save.
  8. Commit the changes in the shopping cart.
  9. Restart the Managed Servers and the Administration Server for the changes to be effective.

    If you are configuring other aspects of ASM in the same configuration change session, you can use a final unique restart to reduce downtime.

About Selecting a Service Migration Policy

When you configure Automatic Service Migration, you select a Service Migration Policy for each cluster. This topic provides guidelines and considerations when selecting the Service Migration Policy.

For example, products or components running singletons or using Path services can benefit from the exactly-once policy. With this policy, if at least one Managed Server in the candidate server list is running, the services hosted by this migratable target are active somewhere in the cluster if servers fail or are administratively shut down (either gracefully or forcibly). This can cause multiple homogenous services to end up in one server on startup.

When you use this policy, you should monitor the cluster startup to identify what servers are running on each server. You can then perform a manual failback, if necessary, to place the system in a balanced configuration.

Other Fusion Middleware components are better suited for the failure-recovery policy.

Based on these guidelines, the following policies are recommended for an Oracle WebCenter enterprise topology:

  • SOA_Cluster: failure-recovery

  • WCC_Cluster: failure-recovery

See Policies for Manual and Automatic Service Migration in Administering Clusters for Oracle WebLogic Server.

Setting the Service Migration Policy for Each Managed Server in the Cluster

After you modify the JTA migration settings for each server in the cluster, you can then identify the services and set the migration policy for each Managed Server in the cluster, using the WebLogic Remote Console:

  1. Log in to the WebLogic Remote Console.
  2. Navigate to the Edit Tree.
  3. In the structure tree, expand Environment > Migratable Targets.
  4. In the Service Migration Policy drop-down menu, select the appropriate policy for the cluster. See About Selecting a Service Migration Policy.
  5. In the Candidate tab, leave the Chosen list box empty. If you do not select any servers from the Available list box, all the available servers in the cluster become candidates for service migration.
  6. Click Save.
  7. Repeat Step 2 to Step 6 for each of the additional Managed Servers in the cluster.
  8. Commit changes in the shopping cart.
  9. Restart the managed servers for the changes to be effective.

    If you are configuring other aspects of ASM in the same configuration change session, you can use a final unique restart to reduce downtime.

Validating Automatic Service Migration
After you configure automatic service migration for your cluster and Managed Servers, validate the configuration, as follows:
  1. If you have not already done so, log in to the WebLogic Remote Console.
  2. Navigate to the Monitoring Tree.
  3. In the structure tree, expand Environment > Migration.
  4. Click Service Migration Data Runtimes.

    The console displays a list of migratable targets and their current hosting server.

  5. In the Migratable Targets table, select a row for the one of the migratable targets.
  6. Note the value in the Migrated To property.
  7. Use the operating system command line to stop the first Managed Server.

    Use the following command to end the Managed Server Process and simulate a crash scenario:

    kill -9 pid

    In this example, replace pid with the process ID (PID) of the Managed Server. You can identify the PID by running the following UNIX command: 

    ps -ef | grep managed_server_name

    Note:

    After you kill the process, the Managed Server might be configured to start automatically. In this case, you must kill the second process using the kill –9 command again.

  8. Watch the terminal window (or console) where the Node Manager is running.

    You should see a message indicating that the selected Managed Server has failed. The message is similar to the following:

    <INFO> <domain_name> <server_name> 
<The server 'server_name' with process id 4668 is no longer alive; waiting for the process to die.>
<INFO> <domain_name> <server_name> 
<Server failed during startup. It may be retried according to the auto restart configuration.>
<INFO> <domain_name> <server_name>
<Server failed but will not be restarted because the maximum number of restart attempts has been exceeded.>
  9. Return to the Oracle WebLogic Remote Console and refresh the table of migratable targets; verify that the migratable targets are transferred to the remaining, running Managed Server in the cluster:
    • Verify that the Migrated to value for the process you killed is now updated to show that it has been migrated to a different host.
    • Verify that the value in the Status of Last Migration column for the process is Succeeded.
  10. Open and review the log files for the Managed Servers that are now hosting the services; look for any JTA or JMS errors.

    Note:

    For JMS tests, it is a good practice to get message counts from destinations and make sure that there are no stuck messages in any of the migratable targets:

    For example, for uniform distributed destinations (UDDs):

    1. Log in to the WebLogic Remote Console and navigate to the Monitoring Tree.

    2. Navigate to Dashboards and click JMS Destinations.

    3. Order by Destination Name and look for the destination.

    4. Review the Messages Current Count and Messages Pending Count values.

Failing Back Services After Automatic Service Migration

When Automatic Service Migration occurs, Oracle WebLogic Server does not support failing back services to their original server when a server is back online and rejoins the cluster.

As a result, after the Automatic Service Migration migrates specific JMS services to a backup server during a fail-over, it does not migrate the services back to the original server after the original server is back online. Instead, you must migrate the services back to the original server manually.

To fail back a service to its original server, use WLST migrate command. For more information, see WLST Command Reference for Oracle WebLogic Server.