Primary-Standby Configuration Issues

Review these troubleshooting tips for commonly encountered primary-standby configuration related issues when working with Oracle Key Vault.

Write Operations Fail in Restricted Mode
When the primary or standby server is in read-only restricted mode, write operations like creation of key fails.
Fast-Start Failover Suspended
Oracle Key Vault displays a Fast-Start Failover suspended error.
How to Verify Primary-Standby Status
After the Primary-Standby nodes pairing, verify the status of the primary-standby nodes.

Parent topic: Troubleshooting Oracle Key Vault

Write Operations Fail in Restricted Mode

When the primary or standby server is in read-only restricted mode, write operations like creation of key fails.

In Primary-Standby configuration, the status of the node on the Oracle Key Vault management console is, Read-Only Restricted.

Probable Cause

The recommended ports are not open between the primary and standby servers.

Solution

Verify the alerts on the Oracle Key Vault management console home page.
Log in to the Oracle Key Vault server through SSH as the support user. Switch to the root user and verify if the network connectivity between the primary and standby servers is successful or not.
Verify ports 1521, 1522, 7443, and 5696.
```
curl -v telnet://ipaddress:port
```
If the command fails, fix the port issue and ensure that the ports are available and open.
Reboot the Oracle Key Vault server.
Verify if the issue is resolved.

Parent topic: Primary-Standby Configuration Issues

Fast-Start Failover Suspended

Oracle Key Vault displays a Fast-Start Failover suspended error.

Example

On Oracle Key Vault Server, you may see the error in /var/log/messages or when you run the show_configuration command from dgmgrl .

ORA-16818: Fast-Start Failover suspended

Probable Cause

An ORA-16818: Fast-Start Failover suspended appears because of a fast start fail over operation failure.

Solution

Gracefully shut down the primary server in a controlled manner using the Power Off option instead of manually turning off the computer. When you power off the Primary Server, you cannot perform a fast start failover function, resulting in an ORA-16818 error.
In a graceful shutdown operation, the primary server's failover status goes into a suspended state with the standby waiting indefinitely for the primary server to be available. This is the expected behavior for a Fast-Start Failover (FSFO) operation.

Note:
Avoid a split-brain scenario, as defined by Oracle Data Guard.
By design, an FSFO operation error occurs only when the primary server shuts down unexpectedly.
If you perform a SHUTDOWN IMMEDIATE or SHUTDOWN NORMAL command in SQL*Plus, then the FSFO does not occur because the database shuts down gracefully.

Parent topic: Primary-Standby Configuration Issues

How to Verify Primary-Standby Status

After the Primary-Standby nodes pairing, verify the status of the primary-standby nodes.

Check the Primary-Standby nodes status from the Oracle Key Vault management console.
1. Log in to the Oracle Key Vault management console.
2. On the Primary-Standby page, go to System, Settings.
3. Check the nodes status.

Check the status from Primary-Standby nodes.

Run these commands to check the status.

/usr/local/dbfw/bin/setup_ha.rb ––status
/usr/local/dbfw/bin/setup_ha.rb ––dg_status
sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --ha_role

Sample Output

$ /usr/local/dbfw/bin/setup_ha.rb --status
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL>
Configuration - DBFWDB

Protection Mode: MaxAvailability
Members:
DBFWDB_HA2 - Primary database
DBFWDB_HA1 - (*) Physical standby database

(*) Fast-Start Failover target

Properties:
FastStartFailoverThreshold = '120'
OperationTimeout = '30'
TraceLevel = 'USER'
FastStartFailoverLagLimit = '30'
CommunicationTimeout = '60'
ObserverReconnect = '30'
FastStartFailoverAutoReinstate = 'TRUE'
FastStartFailoverPmyShutdown = 'FALSE'
BystandersFollowRoleChange = 'ALL'
ObserverOverride = 'FALSE'
ExternalDestination1 = ''
ExternalDestination2 = ''
PrimaryLostWriteAction = 'CONTINUE'

Fast-Start Failover: ENABLED

Threshold: 120 seconds
Target: DBFWDB_HA1
Observer: *******
Lag Limit: 30 seconds (not in use)
Shutdown Primary: FALSE
Auto-reinstate: TRUE
Observer Reconnect: 30 seconds
Observer Override: FALSE

Configuration Status:
SUCCESS

DGMGRL>
Database - DBFWDB_HA1

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 2.00 KByte/s
Active Apply Rate: (unknown)
Maximum Apply Rate: (unknown)
Real Time Query: ON
Instance(s):
dbfwdb

Properties:
DGConnectIdentifier = 'DBFWDB_HA1'
ObserverConnectIdentifier = ''
LogXptMode = 'sync'
RedoRoutes = ''
DelayMins = '0'
Binding = 'optional'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '10'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '0'
TransportLagThreshold = '0'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = ''
LogFileNameConvert = ''
FastStartFailoverTarget = 'DBFWDB_HA2'
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
StaticConnectIdentifier = 'DBFWDB_HA1_DGMGRL'
StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = '%t_%s_%r.dbf'
TopWaitEvents = '(monitor)'

Database Status:
SUCCESS

DGMGRL>
Database - DBFWDB_HA2

Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
dbfwdb

Properties:
DGConnectIdentifier = 'DBFWDB_HA2'
ObserverConnectIdentifier = ''
LogXptMode = 'sync'
RedoRoutes = ''
DelayMins = '0'
Binding = 'optional'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '10'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '0'
TransportLagThreshold = '0'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = ''
LogFileNameConvert = ''
FastStartFailoverTarget = 'DBFWDB_HA1'
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
StaticConnectIdentifier = 'DBFWDB_HA2_DGMGRL'
StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = '%t_%s_%r.dbf'
TopWaitEvents = '(monitor)'

Database Status:
SUCCESS

DGMGRL>

Run these commands to check Oracle Data Guard status. Make sure the Observer is running.

su oracle
dgmgrl /
show configuration;
show database ‘DBFWDB_HA1’;
show database ‘DBFWDB_HA2’;

Sample Output

$ dgmgrl /
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL> show configuration;

Configuration - DBFWDB

Protection Mode: MaxAvailability
Members:
DBFWDB_HA2 - Primary database
DBFWDB_HA1 - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:
SUCCESS (status updated 25 seconds ago)

DGMGRL> show database 'DBFWDB_HA1';

Database - DBFWDB_HA1

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 2.00 KByte/s
Real Time Query: ON
Instance(s):
dbfwdb

Database Status:
SUCCESS

DGMGRL> show database 'DBFWDB_HA2';

Database - DBFWDB_HA2

Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
dbfwdb

Database Status:
SUCCESS

Parent topic: Primary-Standby Configuration Issues