Managing Temporary Outages with a Backup and Redo Failover Strategy

Backup and Redo Failover is a high availability feature that allows protected databases to temporarily direct backups and redo to an alternate Recovery Appliance when the primary Recovery Appliance experiences an outage or requires planned maintenance. This allows protected database backups and redo to continue uninterrupted and preserves complete data protection. It also prevents the local archived log destinations of the database from filling up and impacting the database, which can occur with no alternate backup destination.

Overview of the Backup and Redo Failover Feature

In an environment where Backup and Redo Failover is configured, a protected database sends backups and redo to a primary Recovery Appliance under normal circumstances. When that appliance is unavailable, the protected database sends backups and redo to an alternate Recovery Appliance until service on the primary is restored.

The alternate appliance does not create virtual full backups from the temporary backups it receives; it only stores the backup pieces (incremental and archived log backups). When the primary appliance is back online and operational, the alternate appliance forwards all temporary backups to the primary appliance, which uses them to create the corresponding virtual full backups. After all virtual full backups are created, the protected database resumes sending backups and redo to the primary appliance. The alternate appliance deletes the temporary backup pieces from local storage only after they are successfully forwarded to the primary appliance.

Configuring Backup and Redo Failover

This section explains how to configure Backup and Redo Failover. The basic work flow is as follows:

Configure the primary Recovery Appliance, as described in "Configuring the Primary Recovery Appliance for Backup and Redo Failover".
Configure the alternate Recovery Appliance, as described in "Configuring the Alternate Recovery Appliance for Backup and Redo Failover".
Configure replication from the alternate Recovery Appliance to the primary Recovery Appliance, as described in "Configuring Replication for Backup and Redo Failover".
Configure the protected database to send backups, as described in "Configuring the Protected Database for Backup and Redo Failover".

Configuring the Primary Recovery Appliance for Backup and Redo Failover

To configure the primary Recovery Appliance for Backup and Redo Failover, you perform many of the tasks for setting up a downstream Recovery Appliance in a replication scenario.

Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance

Follow the instructions in "racli add db_user".

For example, log in to the Recovery appliance as root, change to the bin directory, and use the following command to create the VPC user:

# ./racli add db_user --user_name=vpcuser --user_type=vpc

Enter the password for vpcuser user when prompted.

To create the replication user repuser_from_alternate with the CREATE SESSION privilege:

CREATE USER repuser_from_alternate IDENTIFIED BY password;
GRANT CREATE SESSION TO repuser_from_alternate;

The user_name created on the alternate must be the same as the VPC user created on the primary. However, the passwords do not need to be the same.

Task 2 Create a protection policy on the primary Recovery Appliance

Follow the instructions in Creating a Protection Policy. Ensure that the store_and_forward field is set to NO.

For example, execute the following PL/SQL program to create a primary_brf policy:

BEGIN
  DBMS_RA.CREATE_PROTECTION_POLICY (
    protection_policy_name => 'primary_brf',
    description            => 'For protected dbs on primary',
    storage_location_name  => 'delta',
    recovery_window_goal   => INTERVAL '28' DAY,
    guaranteed_copy        => 'NO',
    store_and_forward      => 'NO');
END;

Task 3: Add a database to the protection policy on the primary Recovery Appliance

Follow the instructions in "Enrolling Protected Databases".

For example, execute the following PL/SQL program to add orcl12 to the primary_brf policy that you created in the previous task:

BEGIN
  DBMS_RA.ADD_DB (
    db_unique_name         => 'orcl12',
    protection_policy_name => 'primary_brf',
    reserved_space         => '128G');
END;

Task 4: Grant database access to the VPC user and the replication user on the primary Recovery Appliance

Follow the instructions in "Enrolling Protected Databases".

For example, execute the following PL/SQL programs to grant the VPC user vpcuser and the replication user repuser_from_alternate the required privileges on protected database orcl12:

BEGIN
  DBMS_RA.GRANT_DB_ACCESS (
    username       => 'vpcuser',
    db_unique_name => 'orcl12');
END;
BEGIN
  DBMS_RA.GRANT_DB_ACCESS (
    username       => 'repuser_from_alternate',
    db_unique_name => 'orcl12');
END;

Configuring the Alternate Recovery Appliance for Backup and Redo Failover

To configure the alternate Recovery Appliance, you perform tasks similar to setting up an upstream Recovery Appliance in a replication scenario.

Task 1 Create a protection policy for Backup and Redo Failover on the alternate Recovery Appliance

Follow the instructions in Creating a Protection Policy. Ensure that you set the store_and_forward field to YES.

For example, execute the following PL/SQL program to create an alt_brf policy:

BEGIN
  DBMS_RA.CREATE_PROTECTION_POLICY (
    protection_policy_name => 'alt_brf',
    description            => 'For protected dbs on alternate',
    storage_location_name  => 'delta',
    recovery_window_goal   => INTERVAL '28' DAY,
    guaranteed_copy        => 'NO',
    store_and_forward      => 'YES');
END;

Task 2: Add the database to the protection policy on the alternate Recovery Appliance

Follow the instructions in Enrolling Protected Databases.

For example, execute the following PL/SQL program to add orcl12 to the alt_brf policy that you created in the previous task:

BEGIN
  DBMS_RA.ADD_DB (
    db_unique_name         => 'orcl12',
    protection_policy_name => 'alt_brf',
    reserved_space         => '128G');
END;

Task 3: Grant database access to the VPC user on the alternate Recovery Appliance

You created this user in "Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance".

Follow the instructions in Enrolling Protected Databases.

For example, execute the following PL/SQL program to grant the VPC user vpcuser the required privileges on protected database orcl12 :

BEGIN
  DBMS_RA.GRANT_DB_ACCESS (
    username       => 'vpcuser',
    db_unique_name => 'orcl12');
END;

Configuring Replication for Backup and Redo Failover

After you configure the primary and the alternate Recovery Appliances, you perform tasks similar to setting up replication from the alternate to the primary appliance. In this scenario, the alternate appliance has the upstream role and the primary appliance has the downstream role.

Task 1: Configure an Oracle wallet on the alternate Recovery Appliance

On the alternate Recovery Appliance, use the mkstore utility to create an Oracle auto-login wallet and add the credentials for the replication user you created in "Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance". The alternate Recovery Appliance requires these credentials when it logs in to the primary Recovery Appliance.

To configure an auto-login wallet on the alternate Recovery Appliance:

Run the following command to create an Oracle wallet in the /dbfs_repdbfs/REPLICATION directory:
```
mkstore -wrl /dbfs_repdbfs/REPLICATION -createALO
```
The mkstore utility creates a file named cwallet.sso in the designated location.
Run the following command to add the replication user credentials:
```
mkstore -wrl wallet_location -createCredential serv_name rep_user pwd
```
The placeholders are defined as follows:
- wallet_location is the directory in which you created the wallet in the previous step.
- serv_name is the Oracle network service name that you use in an EZ Connect descriptor to identify the primary Recovery Appliance on the Oracle network.
- rep_user is the user name of the replication user account. This user was created in Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance . The replication user is not created on the alternate.
- pwd is the secure password of the replication user rep_user.
For example, the following command adds credentials for the net service name rapribrf-scan.acme.com using port 1522 and a database name of rapri, and the replication user name repuser_from_alternate:
```
mkstore -wrl /dbfs_repdbfs/REPLICATION -createCredential \
"rapribrf-scan.acme.com:1522/rapri" "repuser_from_alternate" "pwd"
```

Verify that the credentials were properly added for this user by running the following command:

mkstore -wrl /dbfs_repdbfs/REPLICATION -listCredential

Oracle Secret Store Tool : Version 12.1.0.1 Copyright (c) 2004, 2012, Oracle and/or its affiliates. All rights reserved.
List credential (index: connect_string username)
1: rapribrf-scan.acme.com:1522/rapri repuser_from_alternate

The results do not display the password.

Task 2: Create the replication server configuration on the alternate Recovery Appliance

For the primary Recovery Appliance to which this alternate Recovery Appliance will forward backups after an outage, create a replication server configuration by executing DBMS_RA.CREATE_REPLICATION_SERVER.

This task assumes the following:

You want to create a replication server configuration named raprimary_rep.
You want the alternate Recovery Appliance to log in to the primary Recovery Appliance using the replication account repuser_from_alternate. You created this account in "Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance".
The configuration uses the net service name rapribrf-scan.acme.com:1522/rapri that you stored in the Oracle wallet you created in "Task 1: Configure an Oracle wallet on the alternate Recovery Appliance".
The Oracle wallet is stored in /dbfs_repdbfs/REPLICATION.
The file name of the Recovery Appliance Backup Module, which is preinstalled on every Recovery Appliance, is /u01/app/oracle/product/12.1.0.2/dbh1/lib/libra.so. The module functions as an SBT media management library. RMAN references this module when allocating or configuring a channel for backup to the Recovery Appliance (see "Configuring a Protected Database for Recovery Appliance Replication").

To create the replication server configuration:

With SQL*Plus or SQL Developer, connect to the alternate Recovery Appliance metadata database as RASYS.

Run the DBMS_RA.CREATE_REPLICATION_SERVER procedure for the primary Recovery Appliance.

The following example creates the replication server configuration named raprimary_rep for the primary Recovery Appliance:

BEGIN
  DBMS_RA.CREATE_REPLICATION_SERVER (
    replication_server_name => 'raprimary_rep',
    sbt_so_name      => '/u01/app/oracle/product/12.1.0.2/dbh1/lib/libra.so',
    catalog_user_name       => 'RASYS',
    wallet_alias            => 'rapribrf-scan.acme.com:1522/rapri',
    wallet_path             => 'file:/dbfs_repdbfs/REPLICATION');
END;

Confirm the creation of the replication server configuration. The replication_server_name is converted to upper-case and stored as such. Therefore queries with the name should also be upper-case.

For example, run the following query:
```
SELECT COUNT(*) should_be_one 
FROM   RA_REPLICATION_SERVER
WHERE  REPLICATION_SERVER_NAME = 'RAPRIMARY_REP';

SHOULD_BE_ONE
-------------
1
```
If the configuration was created correctly, then the return value is 1.

Task 3: Associate the alternate Recovery Appliance with the protection policy for Backup and Redo Failover

Specify the primary Recovery Appliance to which the alternate Recovery Appliance forwards backups after an outage by assigning the replication server configuration to a protection policy.

This task assumes the following:

You want to use the replication server configuration named raprimary_rep, which you created in "Task 2: Create the replication server configuration on the alternate Recovery Appliance".
You want to add the replication server configuration to protection policy alt_brf, which you created in "Task 1 Create a protection policy for Backup and Redo Failover on the alternate Recovery Appliance".

To associate the replication server configuration with the Backup and Redo Failover protection policy:

Ensure you are connected to the metadata database on the alternate Recovery Appliance as the Recovery Appliance administrator.
Run the DBMS_RA.ADD_REPLICATION_SERVER procedure for the Backup and Redo Failover protection policy and replication server configuration.

For example, execute the following PL/SQL program:
```
BEGIN
  DBMS_RA.ADD_REPLICATION_SERVER (
   replication_server_name => 'raprimary_rep',
   protection_policy_name  => 'alt_brf');
END;
```

Configuring the Protected Database for Backup and Redo Failover

After you configure replication for Backup and Redo Failover, the protected database administrator should perform the tasks in this section so that the protected database can send backups to the primary Recovery Appliance under normal conditions, and to the alternate Recovery Appliance during a planned or unplanned outage.

Task 1: Configure sqlnet.ora to point to the wallet location

Ensure that the sqlnet.ora file contains the location of the Oracle wallet.

The following example shows how the wallet location entry should appear:

SQLNET.WALLET_OVERRIDE = true
WALLET_LOCATION = 
(SOURCE = 
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /u01/app/oracle/product/12.1.0/dbhome_1/dbs/zdlra)
   )
  )

Task 2: Create an auto-login wallet in the location specified in sqlnet.ora

The following example creates an auto-login wallet in the directory specified in "Task 1: Configure sqlnet.ora to point to the wallet location":

$ mkstore -wrl /u01/app/oracle/product/12.1.0/dbhome_1/dbs/zdlra/ -createALO

Task 3: Add the credentials for the primary and alternate Recovery Appliances to the wallet

In this task the protected database administrator adds credentials for the primary and alternate appliances using the VPC user you created in "Task 1: Create a VPC user account and a replication user account on the primary Recovery Appliance" to the wallet.

The following examples add the vpcuser credentials for the primary appliance rapribrf-scan.acme.com:1521/rapri:dedicated and the alternate appliance raaltbrf-scan.acme.com:1521/raalt:dedicated to the wallet on the protected database:

$ mkstore -wrl /u01/app/oracle/product/12.1.0/dbhome_1/dbs/zdlra/ -createCredential "rapribrf-scan.acme.com:1521/rapri:dedicated" "vpcuser" "pwd"
$ mkstore -wrl /u01/app/oracle/product/12.1.0/dbhome_1/dbs/zdlra/ -createCredential "raaltbrf-scan.acme.com:1521/raalt:dedicated" "vpcuser" "pwd"

Task 4: Register the database with the alternate Recovery Appliance and back up the control file

For this task, the protected database administrator performs steps 1 and 2, and the Recovery Appliance administrator performs step 3.

To register the database and back up the control file:

Using RMAN, connect to the protected database as TARGET and to the alternate Recovery Appliance catalog as CATALOG, and then run the REGISTER DATABASE command.
After the REGISTER DATABASE command is completed, back up the current control file to the alternate appliance:
```
BACKUP DEVICE TYPE SBT CURRENT CONTROLFILE;
```
Verify that the control file backup was replicated from the alternate appliance to the primary appliance.

Task 5: Ensure that the database is registered with the primary Recovery Appliance

This step is to confirm that the protected database is registered with the primary appliance. Because replication is configured when the database is registered with the alternate appliance in the previous task, the database should automatically be registered with the primary appliance.

To confirm registration with the primary appliance:

In RMAN, connect to the database using the primary appliance credentials in the CATALOG connect string.

rman TARGET / CATALOG /@rapribrf-scan.acme.com:1521/rapri:dedicated
Run the REGISTER DATABASE command.

The following error should display:
```
RMAN-20002: target database already registered in recovery catalog
```

Note:

The protected database administrator must also create a separate RMAN backup script that directs backups to the alternate Recovery Appliance when the primary appliance is not available, and redirects backups to the primary appliance when it is back in service. This script must connect to the alternate Recovery Appliance catalog and have the CONFIGURE CHANNEL or ALLOCATE CHANNEL command with credential_alias set to the alternate appliance. See Zero Data Loss Recovery Appliance Protected Database Configuration Guide for an example of how to create an RMAN backup script for the Recovery Appliance.
To send real time redo to the alternate Recovery Appliance during the outage of the primary appliance, an additional log archive destination must be defined as an ALTERNATE for the log archive destination used to connect to the primary appliance. The connect string must be defined in the Oracle auto-login wallet, similar to the connect string required for the primary appliance, and using the same VPC user (although the password may be different). See Data Guard Concepts and Administration for an example of how to use the ALTERNATE attribute to automatically fail over to a alternate remote destination.