1. Administrator's Guide
  2. Managing Oracle Key Vault Multi-Master Clusters

5 Managing Oracle Key Vault Multi-Master Clusters

You can create, configure, manage, and administer an Oracle Key Vault multi-master cluster by using the Oracle Key Vault management console.

5.1 About Managing Oracle Key Vault Multi-Master Clusters

You can add or remove nodes from the cluster, disable or enable cluster nodes, and manage activities such as node conflicts and replication.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.2 Creating the First (Initial) Node of a Cluster

To create a cluster, you must convert an existing standalone Oracle Key Vault server to become the first node in the cluster.

This first node is called the initial node. The standalone Oracle Key Vault server can also be a server that has been upgraded to Oracle Key Vault release 18.1 or later from a previous release or can also be the server that is unpaired from a primary-standby configuration. Check Oracle Key Vault Release Notes for known issues about unpair operations and upgrades.

You can use this node to add one or more nodes to the cluster. The node operates in read-only restricted mode until it is part of a read-write pair.

If the Oracle Key Vault server was upgraded from a release earlier than Oracle Key Vault release 12.2 (bundle patch 4) , then generate and activate a new certificate for the node.

  1. Perform a server backup.
  2. Log into the Oracle Key Vault management console as a user who has the System Administrator role.
  3. Select the Cluster tab.
    The Configure as Candidate Node page appears, with the IP address of the current server listed in the Current Server IP field.
  4. On the Configure as Candidate Node page, enter the following information:

    • First Node of Cluster: Select the Yes button.

    • Node Name: Enter a unique name for this node. You cannot change this name after it has been accepted in the name resolution process.

    • Cluster Name: Enter a name for this cluster of nodes. You cannot change this name after it has been accepted in the name resolution process.

    • Cluster Subgroup: Enter a name for this sub-group of nodes, such as a data center name or a logical group name. You cannot change this name after it has been accepted in the name resolution process.

  5. Click Convert to Candidate Node.
After the conversion is complete, the Cluster Management page is displayed and the node is now operating in read-only restricted mode.

Related Topics

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.3 Adding a Node to the Cluster

You can create a read-write pair of nodes or a read-only node.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.3.1 Creating a Read-Write Pair of Nodes in a Cluster

After you create the initial node, you must add an additional read-write peer to the cluster.

You can configure any two nodes as a read-write pair. However, any single node can be read-write paired with only one other node. A node can have one or more read-only nodes connected to it, as long as the total number of nodes in the cluster does not exceed 16.

To create the read-write pair using two nodes in a cluster, you pair a node (referred to as the controller node) with a newly configured server (referred to as the candidate node). Note that this will take some time: an hour or more, depending on the speed of your server, network, and volume of data in the cluster.

  1. Perform a backup of the controller node before continuing.
  2. Ensure that the following network requirements are in place:
    • There is good network connectivity between the servers that host the controller node and the candidate node.
    • The ports that are required for Oracle Key Vault are open in the network firewall. These ports are described in Network Port Requirements.
  3. Log into the controller node Oracle Key Vault management console as a user who has the System Administrator role.
    You can use any existing node, including the first node, that does not have a read-write peer to be the controller for this operation. If necessary, add a read-only node first.
  4. Select the Cluster tab.
  5. Click Add.
  6. In Recovery Passphrase of the Cluster, enter the recovery passphrase.
    This value will be used later when you pair with the candidate node.
  7. Select Yes for Add Node as Read-Write Peer.
  8. Enter the following details under Add Candidate Node Details.

    While you enter these details, do not save any of this information or click the Add Node button until you reach Step 9.

    • Candidate Node ID: Select a unique ID for the candidate node. Remember that after you create this ID, you cannot change it.

    • Candidate Node Name: Enter a unique name of the candidate node. After you create this name, you cannot change it.

    • Cluster subgroup Name for Candidate Node: Enter the sub-group name for the candidate node. You can provide an existing subgroup name. If you provide a subgroup name that does not exist, it will be created. Remember that you cannot change the subgroup of this node after the node joins the cluster.

    • IP Address of Candidate Node: Enter the IP address of the candidate node.

  9. In a new browser window, log into the Oracle Key Vault management console of the candidate node as a user who has the System Administrator role.
  10. Select the Cluster tab.
    The Configure as Cluster Candidate page is displayed.
  11. For First Node of Cluster, select No.
  12. For Recovery Passphrase of the Cluster, enter the recovery passphrase of the cluster that you created earlier for the controller node.
  13. For IP Address of the Controller Node, enter the IP address of the controller node.
  14. In the browser window for the controller node, scroll to the bottom of the screen. Select and copy the entire node certificate.
  15. In the browser window for the candidate node, paste the copied certificate from the controller node into the Certificate of the Controller Node field.
    Check the recovery passphrase, the IP address, and the pasted in certificate very carefully to ensure that you copied it correctly. If there is an error, after you click Convert to Candidate Node, you will need to reinstall Oracle Key Vault.
  16. Click Convert to Candidate Node.
    After the conversion is complete, the screen will refresh and show the certificate of the candidate node. The Adding Candidate Node to Cluster page is displayed. This can take several minutes.
  17. Select and copy the entire candidate node certificate.
  18. In the browser window of the controller node, paste the copied certificate from the candidate node into the Certificate of Candidate Node box.
  19. Click Add Node.
  20. Click OK to confirm in the confirmation dialog box.

    This process will take an hour or more, depending on the speed of your server, network, and volume of data in the cluster. During this time, the network management interface of the Oracle Key Vault will be restarted and you might momentarily get a Server Error 500 on the controller node. On the candidate node, errors may also appear, such as Bad Gateway. The candidate node will restart as part of the induction process. This is normal. During the pairing process, the status of the candidate node will display as PAIRING on all cluster nodes.

    To view the status of any server, view the output on the management console.

    After the candidate node restarts, you can log in to either cluster node to view the cluster status by selecting the Cluster tab. Both nodes will now show as ACTIVE. The candidate node may briefly display that it is in read-only restricted mode after it automatically restarts. This node is now a synchronously paired cluster node and no longer a candidate node. After a node is part of a cluster, the console will display the node name, subgroup name, and cluster name in the top right area of the console header.

Related Topics

Parent topic: Adding a Node to the Cluster

5.3.2 Creating a Read-Only Node in a Cluster

To add a new read-only cluster node, you pair any existing cluster node with a newly configured server.

The existing cluster node is referred to as the controller node, and the newly configured server is referred to as the candidate node. This process will take an hour or more, depending on the speed of your server, network, and volume of data in the cluster.

  1. Perform a server backup before continuing.
  2. Ensure that the following network requirements are in place:
    • There is good network connectivity between the servers that host the controller node and the candidate node.
    • The ports that are required for Oracle Key Vault are open in the network firewall. These ports are described in Network Port Requirements.
  3. Log into the controller node Oracle Key Vault management console as a user who has the System Administrator role.
    You can use any existing node as a controller for this operation.
  4. Select the Cluster tab.
  5. Ensure that Management is selected.
  6. In the Cluster Details section click Add.
  7. In the Add Cluster Details section, enter the cluster recovery passphrase in the Recovery Passphrase of the Cluster field.
    This value will be used later when pairing with the candidate node.
  8. Under Add Candidate Node Details, enter the following information:
    • Add Node as Read-Write Peer: Select No .
    • Node ID: Select a unique ID for the candidate node. Remember that after you create this ID, you cannot change it.
    • Node Name: Enter a unique name of the candidate node. After you create this name, you cannot change it.
    • Cluster Subgroup: Enter the subgroup name for the candidate node. You can provide an existing subgroup name. If you provide a subgroup name that does not exist, then it will be created. Remember that you cannot change the subgroup of this node after the node joins the cluster.
    • Cluster Name: This name is populated automatically.
    • IP Address: Enter the IP address of the candidate node.
    Do not exit this page.
  9. In a new browser window, log into the Oracle Key Vault management console of the candidate node as a user who has the System Administrator role.
  10. Select the Cluster tab.
    The Configure as Cluster Candidate page appears.
  11. For First Node of Cluster, select the No option.
  12. For Recovery Passphrase of the Cluster, enter the same recovery passphrase of the cluster value that was entered for the controller node. This is the same value that was entered on the controller node.
  13. For IP Address of the Controller Node, enter the IP address of controller node.
  14. In the browser window for the controller node, scroll to the bottom of the screen. Select and copy the entire node certificate.
  15. In the browser window for the candidate node, paste the copied certificate from the controller node into the Certificate of the Controller Node box.
    Check the recovery passphrase, the IP address, and the pasted in certificate very carefully to ensure that you copied it correctly. If there is an error, after you click Convert to Candidate Node, you will need to reinstall Oracle Key Vault.
  16. Click Convert to Candidate.
    After the conversion is complete, the screen will refresh and show the certificate of the candidate node. The Adding Candidate Node to Cluster page is displayed. This can take several minutes.
  17. Select and copy the entire candidate node certificate.
  18. In the browser window of the controller node, paste the copied certificate from the candidate node into the Certificate of Candidate Node box.
  19. Click Add Node.
  20. Click OK to confirm in the confirmation dialog box.

    This process will take an hour or more, depending on the speed of your server, network, and volume of data in the cluster. During this time, the Oracle Key Vault console of the controller node will become unresponsive and can display an error such as Server Error 500. On the candidate node, errors may also appear, such as Bad Gateway. The candidate node will restart as part of the synchronization process. This is normal. During the pairing process, the status of the candidate node will display as PAIRING on all other cluster nodes not involved in this pairing process.

    To view the status of any server, view the output on the server console.

    After the candidate node restarts, you can log into either cluster node to view the cluster status by selecting the Cluster tab. Both nodes will now show as ACTIVE. The candidate node may briefly display that it is in read-only restricted mode after it automatically restarts.  This node is now a read-only paired cluster node and no longer a candidate node. After a node is part of a cluster, the console will display the node name, sub-group name, and cluster name in the top right area of the console header.

Related Topics

Parent topic: Adding a Node to the Cluster

5.3.3 Creating an Additional Read-Write Pair in a Cluster

Any node can be read-write paired with only one other node, and there can be multiple read-write pairs in a cluster.

  1. Select a read-only cluster node as the controller node to pair with a new candidate node.
  2. Follow the steps to create a read-write pair of nodes in a cluster.

Related Topics

Parent topic: Adding a Node to the Cluster

5.4 Terminating the Pairing of a Node

On the controller node, you can terminate the pairing process for a new node.

  1. On the controller node, in the Status section of the Adding Candidate Node to Cluster page, click the Abort button.
  2. A dialog with the message Are you sure you want to ABORT the addition of the new node? will appear. Select OK.
    After the pairing process terminates, you will be returned to the Add Node to Cluster page on the controller node.
If you terminate the pairing of a candidate node, then the candidate node is no longer usable in its current state. If you want to use the server as a node, then you must re-image the server with the Oracle Key Vault appliance software. However, if it is early enough in the termination process (before any bundle parts have reached the candidate node from the controller node), then you can terminate the pairing on the candidate node. This returns the candidate node to its standalone state. In this case, you do not have to re-image the candidate node.

Related Topics

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.5 Disabling a Cluster Node

You can temporarily disable a cluster node, which is required for upgrades and maintenance.

However, be aware that a node can only be disabled for a set period of time. When it exceeds that time, it cannot be enabled again. The default maximum disable node duration time is 24 hours, but you can set it for as high as 99 hours.

  1. Log into any cluster Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the Cluster tab.
  3. In the Select Node column, select the checkbox of the node to disable.
  4. Click Disable.
    On the node that is being disabled, the node status will display as DISABLING during the disabling process. The other nodes will display the status for this node as DISABLED. When the disabling process is complete, the node that you disabled displays the DISABLED status.

Related Topics

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.6 Enabling a Disabled Cluster Node

You can enable any cluster node that was previously disabled. You must perform this operation from the disabled node.

  1. As a user who has the System Administrator role, log into the Oracle Key Vault management console of any active node in the cluster.
  2. Select the Cluster tab.
    By default, the Management page should appear.
  3. In the Cluster Details section, note the dates in which the nodes have been disabled.
    Oracle recommends that you enable the nodes in the reverse order in which they were disabled. Otherwise, the enabling action may not be able to complete.
  4. In the Cluster Details section, under Node Name, click the name of the node that was disabled most recently.
    Clicking the node name enables you to log in to this node. You can only enable disabled nodes from the disabled node itself.
  5. On the Management page of the disabled node, select Enable.
    You do not need to check the checkbox of the disabled node in the Cluster Details table.
  6. Repeat this step for each disabled node, from the most recent to the node that was disabled first.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.7 Deleting a Cluster Node

You can permanently delete a node from the cluster.

Deleted nodes cannot be added back to any cluster, not just to the current cluster from which they were deleted. However, you can reinstall the Oracle Key Vault appliance software on this server and add the deleted node to a cluster. All data will be synchronized with the cluster before the node is deleted. A node cannot delete itself.

  1. As a user who has the System Administrator role, on a different node, log into the Oracle Key Vault management console.
    A node cannot delete itself.
  2. Select the Cluster tab.
  3. In the Select Node column, select the checkbox of the node to delete.
  4. Click Delete.
    The node status will display as DELETING. After it is deleted, it will show as DELETED, and later be removed from the cluster management page.

    This action is immediate. The node status will display as DELETING. Do not shut down the deleted server until it no longer shows in the cluster status. However, Oracle recommends that you wait an hour after deleting a cluster node before reusing the node ID of the node that was deleted.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.8 Force Deleting a Cluster Node

You can permanently force delete a node from a cluster that is dead, unresponsive, or has exceeded the maximum disabled node time limit.

Forcefully deleting a node that is still a part of a cluster may cause inconsistency in the cluster. Be aware that if the read-write peer of the node that was forcefully deleted is also removed from the cluster before confirming that all critical data from the forcefully deleted node has reached other nodes, then data loss can result. When you forcefully delete a node, ensure that the node to be deleted has first been shut down. A node cannot be deleted from its own management console. When you must forcefully delete a node, ensure that the node to be deleted has first been shut down. Deleted nodes cannot be added back to the cluster. However, you can reinstall the Oracle Key Vault appliance software on a server and then add the deleted node to a cluster.

  1. On a different node, log into the Oracle Key Vault management console as a user who has the System Administrator role.
    A node cannot delete itself. Oracle recommends that if the node to be deleted has a read-write peer, to force delete the node from its read-write peer.
  2. Select the Cluster tab.
  3. In the Select Node column, select the checkbox of the node to force delete.
  4. Click Force Delete.
    The node status will display as DELETING. After it is deleted, it will show as DELETED, and later be removed from the cluster management page. Oracle recommends that you wait an hour after force deleting a cluster node before reusing the node ID of the node that was deleted.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.9 Managing Replication Between Nodes

You can enable and disable node replication from the Oracle Key Vault management console.

  • Restarting Cluster Services
    While managing replication between nodes, you can restart a node's cluster services when the cluster service status for the node is down.
  • Disabling Node Replication
    You can disable the replication link between the current node and any other node in a cluster.
  • Enabling Node Replication
    You can enable the replication link between the current node and any other node in a cluster.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.9.1 Restarting Cluster Services

While managing replication between nodes, you can restart a node's cluster services when the cluster service status for the node is down.

  1. Log into Oracle Key Vault management console of any cluster node as a user who has the System Administrator role.
  2. Select the Cluster tab, and then Monitoring from the left navigation bar.
  3. In the Node State pane, click the Restart Cluster Services button.

Parent topic: Managing Replication Between Nodes

5.9.2 Disabling Node Replication

You can disable the replication link between the current node and any other node in a cluster.

  1. Log into Oracle Key Vault management console of any cluster node as a user who has the System Administrator role.
  2. Select the Cluster tab, and then Monitoring from the left navigation bar.
  3. Select the nodes for which you want to disable replication.
  4. Click Disable.
  5. Click OK to confirm in the dialog box.

Parent topic: Managing Replication Between Nodes

5.9.3 Enabling Node Replication

You can enable the replication link between the current node and any other node in a cluster.

  1. As a user who has the System Administrator role, log in to the Oracle Key Vault management console of the node for which replication should be managed.
  2. Select the Cluster tab, and then Monitoring from the left navigation bar.
  3. Click Enable.
  4. Click OK to confirm in the dialog box.

Parent topic: Managing Replication Between Nodes

5.10 Cluster Management Information

The Cluster Management page provides a concise overview of the cluster and the status of each node. 

You can also manage the cluster from the cluster details section. When a node is performing a cluster operation it becomes the controller node.

Be aware that because the replication across the cluster takes time, there may be a delay before the Cluster Management page refreshes with the new cluster status. The replication lag in the monitoring page will help estimate the delay.

To view the Cluster Management page, click the Cluster tab, and then Management  from the left navigation bar.

Description of cluster_management.png follows
Description of the illustration cluster_management.png

Current Node Information

  • Node Name: The name of this node.

  • Node Type: The type of node, such as whether it is read-only or read-write.

  • Cluster Subgroup: The subgroup to which this node belongs.

Cluster Information

  • Cluster Name: The name of the cluster.

  • Cluster Subgroups: All subgroups within the cluster.

  • Maximum Disable Node Duration: The maximum time, in hours, that a node can be disabled before it is evicted from the cluster.

  • Cluster Version: The version of Oracle Key Vault in which the cluster is operating.

Cluster Details

  • Select Node: Used to select a node for a specific operation, such as delete, force delete, or disable.

  • Node ID: The ID of the node.

  • Node Name: The name of the node. Clicking the node name takes you to the Cluster Management page of that node.

  • IP Address: The IP address of the node.

  • Mode: The type of node, such as read-write, read-only, or read-only restricted.

  • Status: The status of the node, such as active, pairing, disabling, disabled, enabling, deleting, or deleted.

  • Read-Write Peer: The read-write peer of the node. If blank, it has no read-write peer.

  • Cluster Subgroup: The subgroup to which the node belongs.

  • Join Date: The date and time that the node was added to the cluster.

  • Disable Date: The date and time that the node was disabled.

  • Node Version: The current version of the Oracle Key Vault node.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.11 Cluster Monitoring Information

The Cluster Monitoring page provides the replication health of the cluster and the current node.

This page also provides a concise overview of the settings enabled in the cluster. You cannot update the settings on this page. Because the replication across the cluster takes time, there may be a delay before the Cluster Monitoring page refreshes with the new cluster state. Replication lag will help estimate the delay.

To view the cluster monitoring page, click the Cluster tab, and then Monitoring from the left navigation bar.

You can hover the mouse over the checkmarks or X's in the Cluster Settings State pane. It will display one of the following explanations of the state:
  • Enabled in Cluster
  • Enabled in Node
  • Disabled in Cluster
  • Disabled in Node

Description of cluster-monitoring-information.png follows
Description of the illustration cluster-monitoring-information.png

Cluster Link State

  • Select Node: Used to select nodes for a specific operation, such as enabling or disabling replication. You can click the checkbox on the label row to select all nodes.

  • Node ID: The ID of the node.

  • Node Name: The name of the node.

  • State: The current state of the node. The server is either up or down.

  • Heartbeat Lag: The average time of the heartbeat.

  • Replication Lag: The average time to replicate an object.

  • Enable: Enables the replication between the current node and the node selected.

  • Disable: Disables the replication between the current node and the node selected.

Cluster Settings State

  • Node ID: The ID of the node.

  • Node Name: The name of the node.

  • Audit: Indicates if auditing is enabled or disabled.

  • FIPS: Indicates if FIPS mode is enabled or disabled.

  • HSM: Indicates if HSM integration is enabled or disabled.

  • SNMP: Indicates if SNMP is enabled or disabled.

  • SYSLOG: Indicates if syslog is enabled or disabled.

  • DNS: Indicates if DNS is enabled or disabled.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.12 Naming Conflicts and Resolution

Oracle Key Vault can resolve naming conflicts that can arise as users create objects such as endpoints, endpoint groups, and user groups.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters

5.12.1 About Naming Conflicts and Resolution

If you create an object that has the same name as another object on another node, Oracle Key Vault resolves this conflict.

You can create a new object with a name that conflicts with an object of the same type created on another node. If a conflict happens, then Oracle Key Vault will make the name of the conflicting object unique by adding _OKVxx, where xx is a node number, to the end of the user-supplied name. You can choose to accept this new name or change the object name.

System administrators can resolve the following naming conflicts:

  • User names
  • Endpoint names

Key administrators can resolve the following naming conflicts:

  • Endpoint groups
  • Security objects
  • User groups
  • Wallets

If an object is stuck in the PENDING state and will not transition to ACTIVE, then check for any broken replication links in the cluster. You can find cluster links in the Oracle Key Vault management console by selecting the Cluster tab and then selecting Monitoring.

Parent topic: Naming Conflicts and Resolution

5.12.2 Naming Conflict Resolution Information

The Cluster Conflict Resolution page provides a list of objects with names that conflict with objects created on different nodes. 

On this page, you can accept the suggested unique name or edit the object name. To view the Cluster Conflict Resolution page, click the Cluster tab, and then Conflict Resolution from the left navigation bar. Alternatively, you can click on the Click here for details button on a Naming Conflict alert from the Alerts table on the Home page.


Description of walletconflict.png follows
Description of the illustration walletconflict.png

Wallet Name Conflicts

  • Unique Name: The unique name assigned to the object by the system.

  • Supplied Name: The original name of the object that conflicts with another object of this type.

  • Name Status: The status of the object. The status can be PENDING or ACTIVE.

  • Created By: The user that created the conflicting object name.

  • Creator Node: The node that the conflicting object was created on.

  • Description: The description of the object as entered by the user.

  • Rename: The button that links to the object page where it can be renamed.

  • Accept: Allows you to accept the suggested name for the selected objects.

Parent topic: Naming Conflicts and Resolution

5.12.3 Changing the Suggested Conflict Resolution Name

You can change the suggested name for an object that conflicts with another object of the same type.

  1. As a user who has the appropriate administrator role, log in to the Oracle Key Vault management console.
  2. Select the Cluster tab, and then Conflict Resolution from the left navigation bar.
  3. Locate the object that requires a name change.
  4. Click the edit icon to the right of the object.
  5. On the object overview page, enter the new name for the object.
  6. Click Save.

Parent topic: Naming Conflicts and Resolution

5.12.4 Accepting the Suggested Conflict Resolution Name

You can accept the suggested name for an object name that conflicts with another object of the same type.

  1. As a user who has the appropriate administrator role, log in to the Oracle Key Vault management console.
  2. Select the Cluster tab, and then Conflict Resolution from the left navigation bar.
  3. Select the objects for which you want to accept the suggested name.
  4. Click Accept.

Parent topic: Naming Conflicts and Resolution

5.13 Multi-Master Cluster Deployment Recommendations

Oracle provides deployment recommendations for deployments that have two or more nodes.

Two-Node Deployment Recommendations

Use a two-node deployments for the following situations:

  • Non-critical environments, such as test and development
  • Simple deployment of read-write pairs with both nodes active, replacing classic primary-standby
  • Single data center environments

Considerations for a two-node deployment:

  • Availability is provided by multiple nodes.
  • Maintenance will require down time.
  • Good network connectivity between data centers is mandatory.

Three-Node Deployment Recommendations

Use a three-node deployment for the following situations:

  • Single data center environments with minimal downtime requirement
  • Single read-write pair with additional read-only node to handle load
  • One read-only node is available for zero downtime during maintenance

Considerations for a three-node deployment:

  • Take regular backups to remove destinations for disaster recovery.

Four or More Node Deployment Recommendations

Use a deployment of four or more nodes for the the following situations:

  • Large data centers distributed across geographical locations
  • Deployment of read-write pairs with pair members spanning geography

Considerations for a large deployment:

  • Availability is provided by multiple nodes.
  • Additional read-only nodes can be used to handle load.
  • Good network connectivity between data centers is mandatory.

Parent topic: Managing Oracle Key Vault Multi-Master Clusters