Messaging Server supports Oracle Clusterware high availability solution. Messaging Server provides agents for this solution.

Messaging Server supports different cluster topologies. Refer to the appropriate cluster product documentation for more information. In addition, you can build in availability to your Messaging Server deployment by making infrastructure components, such as Directory Server, highly available.The following topics in this section explain the options available for each component.

In addition to evaluating a purely highly available (HA) solution, you should consider deploying hardware that is capable of ASR.

ASR is a process by which hardware failure related downtime can be minimized. If a server is capable of ASR, it is possible that individual component failures in the hardware result in only minimal downtime. ASR enables the server to reboot itself and configure the failed components out of operation until they can be replaced. The downside is that a failed component that is taken out of service could result in a less performing system. For example, a CPU failure could result in a machine rebooting with fewer CPUs available. A system I/O board or chip failure could result in system with diminished or alternative I/O paths in use.

Different SPARC systems support very different levels of ASR. Some systems support no ASR, while others support very high levels. As a general rule, the more ASR capabilities a server has, the more it costs. In the absence of high availability software, choose machines with a significant amount of hardware redundancy and ASR capability for your data stores, assuming that it is not cost prohibitive.

From a Messaging Server standpoint, the most important factor in planning your directory service is availability. As an infrastructure service, the directory must provide as near-continuous service as possible to the higher-level applications for authorization, access, email routing, and so forth.

A key feature of Directory Server that provides for high availability is replication. Replication is the mechanism that automatically copies directory data from one Directory Server to another. Replication enables you to provide a highly available directory service, and to geographically distribute your data. In practical terms, replication brings the following benefits:

• Failover

• Load balancing

• Higher performance and reduced response times

• Local data management

Table 10-1 shows how you can design your directory for availability.

See the Directory Server Documentation for more information.

You can configure Messaging Server to be highly available by using clustering software. Messaging Server supports Oracle Clusterware software.

In a tiered Messaging Server architecture, where front-end and back-end components are distributed onto separate machines, you would want to make the back-end components highly available through cluster technology as the back ends are the “stores" maintaining persistent data. Cluster technology is not typically warranted on the Messaging Server front ends as they do not hold persistent data. Typically, you would want to make the Messaging Server MTA and MMP, and Webmail Server front ends highly available through redundancy, that is, by deploying multiple front-end hosts. You could also add high availability to the MTA by protecting its disk subsystems through RAID technology.

In addition to the high availability solutions discussed in the previous sections, you can use enabling techniques and technologies to improve both availability and performance. These techniques and technologies include load balancers, Directory Proxy Server, and replica role promotion.

For more information on high availability models, see the following product documentation:

Oracle Clusterware

• Oracle Clusterware Administration and Deployment Guide

Remote site failover is the ability to bring up a service at a site that is WAN connected to the primary site in the event of a catastrophic failure to the primary site. There are several forms of remote site failover and they come at different costs.

For all cases of remote site failover, you need additional servers and storage capable of running all or part of the users' load for the service installed and configured at the remote site. By all or part, this means that some customers might have priority users and non-priority users. Such a situation exists for both ISPs and enterprises. ISPs might have premium subscribers, who pay more for this feature. Enterprises might have divisions that provide email to all of their employees but deem this level of support too expensive for some portion of those users. For example, an enterprise might choose to have remote site failover for mail for those users that are directly involved in customer support but not provide remote site failover for people who work the manufacturing line. Thus, the remote hardware must be capable of handling the load of the users that are allowed to access remote failover mail servers.

While restricting the usage to only a portion of the user base reduces the amount of redundant server and storage hardware needed, it also complicates configuration and management of fail back. Such a policy can also have other unexpected impacts on users in the long term. For instance, if a domain mail router is unavailable for 48 hours, the other MTA routers on the Internet will hold the mail destined for that domain. At some point, the mail will be delivered when the server comes back online. Further, if you do not configure all users in a failover remote site, then the MTA will be up and give permanent failures (bounces) for the users not configured. Lastly, if you configure mail for all users to be accepted, then you have to fail back all users or set up the MTA router to hold mail for the nonfunctional accounts while the failover is active and stream it back out once a failback has occurred.

Potential remote site failover solutions include:

• Simple, less expensive scenario. The remote site is not connected by large network bandwidth. Sufficient hardware is setup but not necessarily running. In fact, it might be used for some other purpose in the meantime. Backups from the primary site are shipped regularly to the remote site, but not necessarily restored. The expectation is that there will be some significant data loss and possibly a significant delay in getting old data back online. In the event of a failure at the primary site, the network change is manually started. Services are started, followed by beginning the imsrestore process. Lastly, the file system restore is started, after which services are brought up.

• More complicated, more expensive solution. Oracle sell software solutions that cause all writes occurring on local (primary) volumes to also be written to remote sites. In normal production, the remote site is in lock step or near lock step with the primary site. Upon primary site failure, the secondary site can reset the network configurations and bring up services with very little to no data loss. In this scenario, there is no reason to do restores from tape. Any data that does not make the transition prior to the primary failure is lost, at least until failback or manual intervention occurs in the case of the MTA queued data. This solution requires a significant increase in the quantity of hardware at the primary site as there is a substantial impact in workload and latency on the servers to run the data copy.

• Most available solution. This solution is essentially the same as the software real time data copy solution except the data copy is not happening on the Message Store server. If the Message Store servers are connected to storage arrays supporting remote replication, then the data copy to the remote site can be handled by the storage array controller itself. Storage arrays that offer a remote replication feature tend to be large, so the base cost of obtaining this solution is higher than using lower-end storage products.

There are a variety of costs to these solutions, from hardware and software, to administrative, power, heat, and networking costs. These are all fairly straightforward to account for and calculate. Nevertheless, it is difficult to account for some costs: the cost of mistakes when putting a rarely practiced set of procedures in place, the inherent cost of downtime, the cost of data loss, and so forth. There are no fixed answers to these types of costs. For some customers, downtime and data loss are extremely expensive or totally unacceptable. For others, it is probably no more than an annoyance.

In doing remote site failover, you also need to ensure that the remote directory is at least as up to date as the messaging data you are planning to recover. If you are using a restore method for the remote site, the directory restore needs to be completed before beginning the message restore. Also, it is imperative that when users are removed from the system that they are only tagged as disabled in the directory. Do not remove users from the directory for at least as long as the messaging backup tapes that will be used might contain those users' data.

The basic asymmetric or hot standby high availability model consists of two clustered host machines or nodes. A logical IP address and associated host name are designated to both nodes.

In this model, only one node is active at any given time; the backup or hot standby node remains idle most of the time. A single shared disk array between both nodes is configured and is mastered by the active or primary node. The message store partitions and MTA queues reside on this shared volume.

Figure 10-1 shows two physical nodes, Physical-A and Physical-B. Before failover, the active node is Physical-A. Upon failover, Physical-B becomes the active node and the shared volume is switched so that it is mastered by Physical-B. All services are stopped on Physical-A and started on Physical-B.

The advantage of this model is that the backup node is dedicated and completely reserved for the primary node. Additionally, there is no resource contention on the backup node when a failover occurs. However, this model also means that the backup node stays idle most of the time and this resource is therefore under utilized.

The basic symmetric or "dual services" high availability model consists of two hosting machines, each with its own logical IP address. Each logical node is associated with one physical node, and each physical node controls one disk array with two storage volumes. One volume is used for its local message store partitions and MTA queues, and the other is a mirror image of its partner's message store partitions and MTA queues.

Figure 10-2 shows the symmetric high availability mode. Both nodes are active concurrently, and each node serves as a backup node for the other. Under normal conditions, each node runs only one instance of Messaging Server.

Upon failover, the services on the failing node are shut down and restarted on the backup node. At this point, the backup node is running Messaging Server for both nodes and is managing two separate volumes.

The advantage of this model is that both nodes are active simultaneously, thus fully utilizing machine resources. However, during a failure, the backup node will have more resource contention as it runs services for Messaging Server from both nodes. Therefore, you should repair the failed node as quickly as possible and switch the servers back to their dual services state.

This model also provides a backup storage array. In the event of a disk array failure, its redundant image can be picked up by the service on its backup node.

To configure a symmetric model, you need to install shared binaries on your shared disk. Note that doing so might prevent you from performing rolling upgrades, a feature that enables you to update your system during Messaging Server patch releases.

The N + 1 or "N over 1" model operates in a multi-node asymmetrical configuration. N logical host names and N shared disk arrays are required. A single backup node is reserved as a hot standby for all the other nodes. The backup node is capable of concurrently running Messaging Server from the N nodes.

Figure 10-3 illustrates the basic N + 1 high availability model.

Upon failover of one or more active nodes, the backup node picks up the failing node's responsibilities.

The advantages of the N + 1 model are that the server load can be distributed to multiple nodes and that only one backup node is necessary to sustain all the possible node failures. Thus, the machine idle ratio is 1/N as opposed to 1/1, as is the case in a single asymmetric model.

To configure an N+1 model, you need to install binaries only on the local disks (that is, not shared disks as with the symmetric model). The current Messaging Server installation and setup process forces you to put the binaries on the shared disk for any symmetric, 1+1, or N+1 asymmetrical or symmetrical HA solution.

For information about installing Oracle Clusterware, see the overview of installing Oracle Clusterware in the Oracle Clusterware Administration and Deployment Guide.

After Oracle Clusterware is installed:

1. On each node in the cluster, including the NFS server (if used), create the Messaging Server runtime user and group under which the Messaging Server will run.

   The user ID and group ID numbers must be the same on all nodes in the cluster. The run time user ID is the user name under which Messaging Server runs. This name should not be root. The default is mailsrv. The runtime Group ID is the group under which Messaging Server runs. The default is mail. Although the init-config utility can create these names for you, you can also create them before running configure as part of the preparation of each node as described in this chapter. The runtime user and group ID names must be in the following files:

   • mailsrv, or the name you select, must in /etc/passwd on all nodes in the cluster

   • mail, or the name you select, must be in /etc/group on all nodes in the cluster

2. Configure NFS shares with proper options on the NFS server machine and export them to the NFS clients (all Cluster nodes). Also make sure that NFS shares are on highly available storage. To use file systems other then NFS, like Failover File system or Clustered File systems created on shared storage, refer to Oracle Clusterware documentation.

   For NFS, mount all the NFS shares on all cluster nodes. These NFS mounts will be used for installing Messaging Server binaries, keeping configuration and runtime data.

   If a two-node symmetric cluster setup used, then two NFS mounts are needed on both nodes for Messaging Server instance 1 on node1 and Messaging Server instance 2 on node 2. Here is an example of the NFS share details from /etc/exportfs file on Linux.

   Node 1 : /export/msg1 <NFS Client1 IP > (rw,nohide,insecure,no_subtree_check,async,no_root_squash)
   
   Node 2: /export/msg2 <NFS Client2 IP > (rw,nohide,insecure,no_subtree_check,async,no_root_squash) - (in case of Symmetric HA)
   

   On all cluster nodes, mount the NFS shares with following options. To make the mount points persistent across the boots, keep mount details in the /etc/fstab file.

   Node 1:/export/msg1 /export/msg1 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600
   
   Node 2:/export/msg2 /export/msg2 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 (In case of symmetric HA)
   

3. Create an HA logical host name resource and bring it on-line.

   /u01/app/12.1.0/grid/bin $ ./appvipcfg create -network=1 -ip=<logical IP> -vipname=<Logical IP Resource Name> -user=root
   

   For example:

   /u01/app/12.1.0/grid/bin $ ./appvipcfg create -network=1 -ip=10.0.0.3 -vipname=msg1 -user=root
   

4. Install the required Messaging Server packages on the primary node. Use the Messaging Server installer to install the Messaging Server packages.

   For asymmetric deployments: For NFS, install Messaging Server binaries on local file systems OR NFS mounts on each node of Oracle Clusterware. Install configuration data and run time data only on NFS mounted directory. For example, on /export/msg1.

   For symmetric deployments: For NFS, install Messaging Server binaries and configuration data on NFS mounts on node of Oracle Clusterware. For example, for Messaging Server instance 1 binaries and the configuration data on /export/msg1 and Messaging Server instance 2 binaries and the configuration data on /export/msg2.

5. Configure the Messaging Server.

   In the initial runtime configuration, you are asked for the Fully Qualified Host Name. You must use the HA logical hostname instead of the physical hostname.

6. For legacy configuration, run the HAConfig.rcp recipe to set service.listenaddr and to configure the dispatcher.cnf and job_controller.cnf for high availability.

   The script ensures that the logical IP address is set for these parameters and files, rather than the physical IP address. It also enables the watcher process (sets local.watcher.enable to 1), and the auto restart process (local.autorestart to 1).

   The HAConfig.rcp recipe should only be run once on the primary node.

7. Create an HA Messaging Server resource and start the resource.

   It is now time to create the HA Messaging Server resource. This resource is dependent upon the HA logical host name and HA disk resource if NFS mounts are not used (Eg. Cluster File system).

   ~ $ /u01/app/12.1.0/grid/bin/crsctl add type ocucs.ms.type -basetype cluster_resource -attr "ATTRIBUTE=INSTANCE_PATH,TYPE=string,FLAGS=READONLY|REQUIRED"
   
   ~ $ /u01/app/12.1.0/grid/bin/crsctl add resource ocucs.ms.msg1 -type ocucs.ms.type -attr " INSTANCE_PATH=<instance_path>/messaging64, AGENT_FILENAME='%CRS_HOME%/bin/scriptagent', ACTION_SCRIPT='<instance_path>/messaging64/cwha/bin/ms_actionscript.pl', ENABLED=1, AUTO_START=restore, UPTIME_THRESHOLD=10m, CHECK_INTERVAL=10, SCRIPT_TIMEOUT=300, RESTART_ATTEMPTS=2, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES='hard(msg1) pullup(msg1)', STOP_DEPENDENCIES='hard(intermediate:msg1)',CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*,PLACEMENT=favored"
   
   /u01/app/12.1.0/grid/bin $ /u01/app/12.1.0/grid/bin/crsctl stop resource msg1
   
   /u01/app/12.1.0/grid/bin $ /u01/app/12.1.0/grid/bin/crsctl start resource ocucs.ms.msg1 -n cl1
   

   Where ocucs.ms.type is the Messaging Server resource type, ocucs.ms.msg1 is the Messaging Server resource name, and cl1 is the primary node.

   For a Symmetric HA setup, you should create another Messaging Server resource on the primary node for the second Messaging Server installation.

To unconfigure the high availability components for Oracle Clusterware.

1. Stop the Messaging server resource ocucs.ms.msg1.

   /u01/app/12.1.0/grid/bin/crsctl stop resource ocucs.ms.msg1
   

2. Remove the Messaging Server resource ocucs.ms.msg1.

   /u01/app/12.1.0/grid/bin/crsctl delete resource ocucs.ms.msg1
   

3. Stop the HA logical IP resource msg1.

   /u01/app/12.1.0/grid/bin/crsctl stop resource msg1
   

4. Remove the HA logical IP resource msg1.

   /u01/app/12.1.0/grid/bin/crsctl delete resource msg1
   

5. Remove the Messaging server resource type ocucs.ms.type.

   /u01/app/12.1.0/grid/bin/crsctl delete type ocucs.ms.type
   

6. Uninstall the Messaging Server.

7. Repeat the steps above for each instance in the Cluster setup if more then one instance is present.