You can set up multiple instances of ASAP for high availability and upgrade them in succession with zero down-time. With multiple independent ASAP instances, you can distribute and balance the load. You can also deploy new ASAP instances with new cartridges in the same environment without down time.

You can set up your highly available ASAP environment using virtual machines. The database could be centrally located, with independent table spaces for each ASAP instance.

When you set up multiple ASAP instances, when an instance goes down, the incoming orders are routed to the other available running instances in the environment. Tasks such as routing the requests to the corresponding ASAP instance queue and monitoring the ASAP instance states are performed by Order Balancer. Order Balancer stores the Order ID-to-instance mapping information in its database. Based on the work order ID, a change order or a rollback order would be routed to the instance that initially processed that order.

Order Balancer primarily consists of Order Balancer Queue, a queue where the requests from upstream systems are sent to. Order Balancer routes the requests to the registered ASAP instances and monitors the status of the instances.

For list of supported sticky and non-sticky requests, see "Sticky and Non-Sticky Requests Supported by Order Balancer".

The ASAP instances that you want to register with Order Balancer should have the same configuration and cartridges. Any instance can service any non-sticky request.

Oracle recommends that you create an SAF agent between Order Balancer, all ASAP WebLogic Servers, and the remote application WebLogic Server. Oracle recommends this SAF agent for JMS to ensure reliable communication.

Figure 6-1 illustrates SAF configuration between Order Balancer with multiple ASAP instances and a client on a remote application. In this illustration, OSM is used as an example of a remote application.

Figure 6-1 Connecting Order Balancer with Multiple ASAP Instances to a Remote Application

Description of "Figure 6-1 Connecting Order Balancer with Multiple ASAP Instances to a Remote Application"

In this example, an OSM SAF agent sends requests to the Order Balancer request queue. Based on the order type in the requests, Order Balancer routes the requests to the message queue of the corresponding ASAP member instance. The ASAP instance processes the requests and returns responses through the ASAP SAF agent to the OSM reply-to queue. In case of exceptions in Order Balancer, such as ASAP server is down, and invalid order sent in the request, Order Balancer sends the error responses from Response Queue with an SAF agent to OSM reply-to-queue. Each ASAP member instance sends work order state changes from their respective JSRP XVTEventTopic through a JMS bridge with an SAF agent to the OSM event queue.

For detailed instructions on creating SAF and JMS bridges between ASAP and OSM, see the Configuring WebLogic Resources for OSM Integration With ASAP And UIM On Different Domains (Doc ID 1431235.1) knowledge article on My Oracle Support. This article is applicable to any remote application that uses a WebLogic JMS server to send and receive web service messages or JMS messages.

You can set up and run multiple Order Balancer instances for high availability. Each Order Balancer handles the requests for all the registered ASAP instances. For example, if two independent Order Balancer instances are running, Order Balancer OB1 fails, the other Order Balancer OB2 distributes the requests to all the registered ASAP instances. All the Order Balancer instances use the same database.

When you pause the ASAP instances for maintenance activities such as ASAP upgrade, you must wait until the CACHE_EXPIRY interval expires after the pause request and before proceeding with the maintenance activity on ASAP for the changes to reflect in all the Order Balancer instances.

You can plan your installation based on the available resources and your business requirements.

WebLogic Server

You can install Order Balancer on an existing ASAP WebLogic Server or on a separate WebLogic Server. If you choose to install Order Balancer on an existing ASAP WebLogic Server, install it in a separate domain. If you choose to use a separate WebLogic Server, install WebLogic Server and create a domain for Order Balancer.

Database User

You can use the same database as the ASAP database or a different database. In both the cases, create a separate tablespace and a user.

For example:

create tablespace "OBDATA_TS" logging datafile '/u01/app/oracle/oradata/ASAP72/OBDATA_TS1.dbf' size 31999M reuse autoextend on next 5120K maxsize unlimited default storage (initial 256K next 256K minextents 1 maxextents 2147483645 pctincrease 0);

CREATE USER "OB_SYS_USR" PROFILE "DEFAULT" IDENTIFIED BY "OB_SYS_USR" DEFAULT TABLESPACE "OBDATA_TS" TEMPORARY TABLESPACE "TEMP" ACCOUNT UNLOCK;

GRANT UNLIMITED TABLESPACE TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT "CONNECT" TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT "RESOURCE" TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT CREATE VIEW TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT EXECUTE ON SYS.UTL_FILE TO "OB_SYS_USR" WITH GRANT OPTION; 
GRANT CREATE PROCEDURE TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT CREATE TRIGGER TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT CREATE TABLE TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT CREATE TYPE TO "OB_SYS_USR" WITH ADMIN OPTION; 
GRANT CREATE SEQUENCE TO "OB_SYS_USR" WITH ADMIN OPTION;
GRANT CREATE SESSION TO "OB_SYS_USR" WITH ADMIN OPTION;

For more details about creating tablespaces, users, and granting permissions, see "Creating Oracle Database Tablespaces, Tablespace User, and Granting Permissions".

You install Order Balancer by running the Order Balancer installer. The installer is a command line interface tool, which connects and deploys the Order Balancer application (ASAPOrderBalancer.ear) to the WebLogic Server and creates the required components such as a JMS server, JDBC resources, and wallet files. You run the installer in silent mode.

installOB -properties config.properties -d

Table 6-1 lists the configuration properties that you should specify in the sampleConfig.properties file for installing Order Balancer.

Table 6-2 lists the optional configuration parameters that you specify for installing Order Balancer.

Order Balancer queues enable you to route the requests from upstream systems to Order Balancer, which distributes the requests among the ASAP instances for high availability. Table 6-3 describes the Order Balancer queues that the installer configures automatically.

Order Balancer provides the Web Services interface through which external applications can manage service activation activities and operations. The supported external transport protocols are HTTP and HTTPS and the data service formats are SOAP v1.1 and 1.2. Order Balancer Web Services are of type JAX-RPC.

When routing a request to an ASAP member instance queue, Order Balancer attempts to connect to the instance queue using the connection factory details. If the connection attempt fails in the first attempt, Order Balancer reattempts the connection after the configured time interval. If the connection attempt fails again, Order Balancer marks the instance as down and sets the instance status to DOWN. If the request is a non-server dependent request, Order Balancer attempts to route the request to the next available instance.

Once an instance is marked as down, Order Balancer polls the instance for its status at the configured interval. After Order Balancer makes a successful connection to the member instance, the instance status is set to UP.

When a member instance is marked as down, no further requests are routed to it until it is marked back to UP. When a sticky type of request comes in, but the corresponding instance is down, then an error response with serverDownException type is sent to the Order Balancer's response queue after two attempts. When all the instances are down, all the requests are blocked by Order Balancer until at least one instance becomes available or till time-out. An instance becomes available (state set to UP) when Order Balancer successfully makes a connection in its periodical attempts.

Use the ALL_SERVERS_DOWN_WAIT_INTERVAL parameter to configure the duration of the time out interval. In case of time-out before at least one member instance becomes available, the request is forwarded to the error queue.

You can add a new member instance to Order Balancer or remove an existing member instance from Order Balancer. When you add a member instance to Order Balancer, the instance is registered to Order Balancer. Upon successful addition, Order Balancer will start routing subsequent orders to the ASAP member instance.

The scripts for managing member instance are located in installer_location/ob/scripts directory.

For all the scripts mentioned below, the common Order Balancer WebLogic Server properties are taken from the ob/scripts/OB.properties file saved by the installer after successful installation.

when routing the order requests, Order Balancer connects to the ASAP instance using the respective ASAP user ID that is provided while adding ASAP instance. The ASAP user ID must be part of ASAP_WS_USER_GROUP for accessing Web Services and ASAP_Operators or Administrators for accessing ASAP JMS services. You can provide this ASAP user ID that is part of these two groups while adding the ASAP instance to Order Balancer. Order Balancer routes the upstream order requests to the registered ASAP member instances using the ASAP Web Services interface or the ASAP JMS queue based on the incoming channel.

addASAPServer -asapSrvName asap_Srv_Name -asapSrvURL asap_Srv_URL -asapSrvUser asap_Srv_User -asapSrvRequestQueue "asap_Srv_Request_Queue"

removeASAPServer -asapSrvName asap_Srv_Name

migrateASAPServer -migrateFromSrvName migrate_From_Srv_Name -migrateToSrvURL migrate_To_Srv_URL [-migrateToSrvUser migrate_To_Srv_User] [-migrateToSrvRequestQueue migrate_To_Srv_Request_Queue]

Order Balancer periodically checks the status of all the member instances that are down. When a member instance is up, Order Balancer adds it back to the round-robin distribution.

When you run the script for resuming an instance, the instance does not resume if the instance is marked as Pause_down. Order Balancer marks an instance as Pause_down if the instance is down. Order Balancer does not route requests to this instance until it is resumed, but it polls for the instance state. When the instance is restored, Order Balancer changes the state to previous state PAUSE-NEW. You can then resume the instance.

Order Balancer saves order details in its database table for every ASAP instance that is registered. When Order Balancer receives a sticky order, it checks the saved data to determine the correct ASAP instance to route the order to. For a green-field ASAP instance (ASAP instance that did not process any order before), Order Balancer has all the order related data corresponding to the ASAP instance.

For a non-green field ASAP instance, ASAP contains order data in its database tables. When an instance is added to Order Balancer, Order Balancer will not have any ASAP data corresponding to the previous orders sent to ASAP. If a sticky order dependent on the previous orders is sent to ASAP directly, Order Balancer cannot route it to the corresponding ASAP instance unless the ASAP order data is copied to Order Balancer. All the orders should be copied from the ASAP database to the Order Balancer database when it is added to Order Balancer.

After adding an ASAP instance to Order Balancer, orders could be sent to Order Balancer and Order Balancer routes the orders accordingly. In some unpredicted cases, orders could also be sent to ASAP directly while it is still registered with Order Balancer, resulting in mismatched data between ASAP and Order Balancer. The data in the ASAP database and Order Balancer must be in sync when they are found to be out-of-sync after some orders are processed by Order Balancer and some by ASAP directly. Only those orders that are not present in Order Balancer need to be copied from ASAP to Order Balancer.

Prerequisites for Importing Order Data

To import order data from the ASAP database to Order Balancer, ensure that the ASAP instance that is registered with Order Balancer is in the PAUSE-ALL state.

Running the Script to Import Order Data in ASAP to Order Balancer

To import order data in ASAP to Order Balancer, run the following script:

importASAPData -properties <import.properties>

You import data any time after adding an ASAP instance to Order Balancer. It is recommended to import ASAP data right after it is added to Order Balancer using the addASAPServer script.

In case of errors during the import operation, the import process stops. You can run it again. The script ignores the data that is successfully imported in the failed run and imports only the remaining orders.

You can purge Order Balancer routing data from the database. You can purge all the order data with order date before a specific date.

To purge data from the database, run the installer_location/ob/scripts/purgeDbData script.

purgeDbData -dbHost DBHost -dbPort DBPort -dbServiceName ServiceName -purgeAllBeforeDate DD-MM-YYYY

The Order Balancer installer logs are stored in the logs directory in the installer location. You can change the installer logging level by modifying the config/logging.properties file.

The Order Balancer application logs are added to the WebLogic Server log file. The logs are also added to an additional Order Balancer log file Domain_Root/ASAP_OB/logs.

You can use REST APIs to view and update data about the ASAP member instances in Order Balancer.

When you deploy Order Balancer using the configuration properties file, a subset of properties are copied to the obConfig.properties file. Order Balancer uses these properties during runtime. On the successful deployment of Order Balancer, the obConfig.properties file is copied to the Domain_Home/servers/AdminServer/upload/ASAPOrderBalancer/7.4.0.0.0/plan/AppFileOverrides/ASAPOrderBalancer directory. Order Balancer reads this file using the generic file loading overrides. For more information about the overrides, see "Generic File Loading Overrides" in Oracle Fusion Middleware Deploying Applications to Oracle WebLogic Server.

To change the behavior of Order Balancer, you can modify the properties in the obConfig.properties file. After modifying the properties, redeploy Order Balancer for your changes to take effect. For example, to increase the time interval between the instance health polls, in the obConfig.properties file, increase the value of SERVER_POLL_INTERVAL and redeploy or restart the Order Balancer application.

Issue: Error while saving order details to the database.

While Order Balancer attempts to save order processing details to the database, it is possible that some orders return errors and are not saved to the database.

Resolution: Order Balancer saves the details of the failed entries to the ASAPOB_DB_Fallbackxxx.log file located in the OB_Domain/oracle_communications/asap/ directory. This log file contains details of the failed entries, which you can use for correction.

The failed entries are saved to the log file in the following format:

SEVERE: Database persist failed for entry : OrderID,InstanceID,TimeStamp

Issue: Invalid Message Type

Order Balancer does not validate the incoming request message XML, but only parses and fetches the Order ID and the request type. Order Balancer extracts the request type from the message and compares it with the expected types. In case of an unknown or unsupported request type, Order Balancer sends a response with an exception to the replyTo queue that is configured in the request.

The exception that Order Balancer sends is: Exception: RemoteException

The message that Order Balancer displays is: Message: "Unsupported order request:…"

Issue: An exception response while invoking the Web Services Operation.

FaultCode [http://schemas.xmlsoap.org/soap/envelope/Client. Authentication] FaultString [Failed to receive message weblogic.wsee.util.AccessException: The server at https://ASAP1:1234/<ENVID>/Oracle/CGBU/Mslv/Asap/Ws/Http returned a 403 error code (Forbidden)

Resolution: The ASAP member instance user does not have the required User Groups to route the Web Services request. Add the ASAP member instance user to be part of ASAP_WS_USERS_GROUP and ASAP_Operators/Administrator groups. For more information on the Web Services user, see "Security".

Issue: work:WorkContext Element appears in SOAP Response Header while invoking the Order Balancer Web Services.

FaultCode [http://schemas.xmlsoap.org/soap/envelope/Client. Authentication] FaultString [Failed to receive message weblogic.wsee.util.AccessException: The server at https://ASAP1:1234/<ENVID>/Oracle/CGBU/Mslv/Asap/Ws/Http returned a 403 error code (Forbidden)

Resolution: Set the system property weblogic.wsee.workarea.skipWorkAreaHeader to True, to not send work area header in the SOAP messages. The property value is False by default.

This WorkContext is not part of the WSDL, but it is a part of the response message. You can remove it using a handler. There are two options to solve this issue:

1. Use a server handler to remove the header before it passes to the client.
2. Do not use the version in the application

If these options are not workable, you can introduce a system flag to not add this SOAP header. Use the system flag with caution because you may lose any WorkContext information in the SOAP message.