To perform this upgrade, you must upgrade each multi-master cluster node.

For Oracle Key Vault 21.11 you need to upgrade to 21.x. If you are using the version earlier than 21.x, make sure to upgrade the version to 21.x first before proceeding with the version update to 21.11.

The upgrade process involves performing the upgrade on each multi-master cluster node. After you have begun a cluster upgrade, ensure that you upgrade all the nodes in the cluster one after the other, without too much intervening time between upgrades of two nodes.

Upgrading an Oracle Key Vault multi-master cluster includes upgrading each cluster node to the new later version. You must upgrade all nodes to the same Oracle Key Vault version. You should first upgrade the read-only nodes of the cluster, and then upgrade the read-write pairs. As each cluster node is upgraded, its node version is updated to the new version of the Oracle Key Vault. After you complete the upgrade of all cluster nodes, the cluster version is updated to the new version of the Oracle Key Vault. You can check the node version or the cluster version by selecting the Cluster tab, then in the left navigation bar, selecting Management. Oracle Key Vault multi-master cluster upgrade is considered complete when node version and cluster version at each cluster node is updated to the latest version of Oracle Key Vault.

Before you perform the upgrade, note the following:

• Perform the entire upgrade process on all multi-master cluster nodes, without interruption. That is, after you have started the cluster upgrade process, ensure that you try and upgrade all nodes, individually one after the other or in read-write pairs. Do not perform any critical operations or make configuration changes to Oracle Key Vault until you have completed upgrading all the nodes in your environment.
• Be aware that you cannot use any new features that were introduced in this release until you have completed upgrading all of the multi-master cluster nodes. An error is returned when such features are used from the node that has been upgraded. Oracle recommends that you plan the upgrade of all cluster nodes close to each other to ensure availability of the new features sooner.
• Starting in Oracle Key Vault release 21.2, expiration alerts for deactivated or destroyed objects are not generated. If you are upgrading from Oracle Key Vault release 21.1 or earlier, then the following behavior is expected:
  • As each cluster node is upgraded, Oracle Key Vault deletes all expiration alerts for any certificate and secret objects, as well as for key objects that have been revoked or destroyed.
  • Cluster nodes that have not been upgraded yet will continue to generate alerts for these same objects, and also send email notifications for these alerts. This behavior that results in deletion and recreation of alerts may repeat until the last cluster node is upgraded.
  • After the upgrade is complete, expiration alerts for the certificate and secret objects will have the alert type of Certificate Object Expiration and Secret Object Expiration, respectively.

Similar to a standalone or primary-standby environment, you must perform pre-upgrade tasks such as backing up the Oracle Key Vault server.

1. In the server where Oracle Key Vault is installed, log in as user support, and then switch to the root user.
2. Back up the server so that you can recover data in case the upgrade fails.
3. Ensure that no full or incremental backup jobs are running. Delete all scheduled full or incremental backup jobs before the upgrade.
4. Ensure that the server meets the minimum disk space requirements for an upgrade. For example, 6 GB of free space in the /usr/local/dbfw/tmp directory. See the Oracle Key Vault Readme for this release to determine the disk space requirements for the upgrade.
5. Ensure that you disable diagnostics and clean up disk space in /usr/local/dbfw/tmp before you upgrade by performing the following steps:

   If the Oracle Key Vault system being upgraded is from release 21.6 or later, log in to the Oracle Key Vault management console as a user with the System Administrator role, and navigate to the System tab, and then click the Diagnostics button.

   If the Diagnostics Package Files pane is displayed, then click Clear to disable diagnostics. Note that the Diagnostics Package Files pane will be displayed only if the diagnostics bundle was previously generated, and the files were not cleared.

   If the Diagnostics Package Files pane is not displayed, or if the diagnostics bundle was previously generated using the dbfw-diagnostics-package.rb utility, then log in to the Oracle Key Vault system and run the following commands to disable diagnostics and clean up disk space in /usr/local/dbfw/tmp:

   1. SSH into the Oracle Key Vault system as user support, then switch to user root:

      ssh support@<OKV_IP_Address>
      su – root
      

   2. Delete the generated diagnostics zip file and remove the package using the following commands:

      /usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --clean
      /usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --remove

6. Plan for downtime according to the following specifications:

   Oracle Key Vault Usage	Downtime required
   Wallet upload or download	NO
   Java Keystore upload or download	NO
   Transparent Data Encryption (TDE) direct connect	YES (NO with persistent cache)
   Primary Server Upgrade in a primary-standby deployment	YES (NO with persistent cache)

   The Oracle Key Vault client software is backward-compatible. Older Oracle Key Vault client software versions are completely functional with the upgraded Oracle Key Vault server, the new features of Oracle Key Vault are only available with the current client software. TDE-enabled databases before 21c need to restart to load the updated Oracle Key Vault PKCS#11 library that is deployed as part of the Oracle Key Vault client software upgrade. Oracle Database 21c and later can dynamically load the new Oracle Key Vault PKCS#11 library, without downtime. Database endpoints can be upgraded in parallel to reduce total downtime.

7. If the Oracle Key Vault system has a syslog destination configured, ensure that the remote syslog destination is reachable from the Oracle Key Vault system, and that logs are being correctly forwarded. If the remote syslog destination is not reachable from the Oracle Key Vault system, then the upgrade process can become much slower than normal.
8. Check the disk size before you begin the upgrade. If any of the nodes in question have a disk size that is greater than 2 TB and uses BIOS boot mode, then you cannot upgrade that system to the new release. Oracle recommends that you remove the node from the cluster and if possible, replace it with a node whose disk is less than 2 TB in size.
9. If you need to increase available disk space, then remove the temporary jar files located in /usr/local/okv/ssl. Be careful in doing so. If you accidentally delete any files other than the jar files in /usr/local/okv/ssl, then the Oracle Key Vault server becomes non-functional.
10. Check the boot partition size. If any of the nodes in question have a boot partition that is less than 500 MB, then you cannot upgrade that system to the new release. You can check this size as follows:
    1. Mount the /boot partition.

       /bin/mount /boot

    2. Check the Size column given by the following command:

       /bin/df -h /boot

    3. Unmount the /boot partition:

       /bin/umount /boot

    If the boot partition given by this command shows less than 488 MB, then you cannot upgrade to the current release. Oracle recommends that you remove the node from the cluster and if possible, replace it with a node that has been freshly installed with the same Oracle Key Vault version as the rest of the cluster nodes.
11. Increase the Maximum Disable Node Duration setting as appropriate so that any disabled cluster nodes have sufficient time to be upgraded then enabled back into the cluster. Note that increasing the Maximum Disable Node Duration setting also increases disk space usage.
12. Plan to disable one node at a time.
13. If Oracle Audit Vault was integrated with Oracle Key Vault in Oracle Key Vault release 21.2 or earlier, then do the following to disable and remove the Oracle Audit Vault integration:
    1. Disable the Oracle Audit Vault integration: Log into the Oracle Key Vault management console as a System Administrator, select the System tab and then Settings from the left navigation bar. In the Monitoring and Alerts pane, select Audit Vault. In the Audit Vault integration pane that appears, disable Oracle Audit Vault. Click Save.
    2. Log in to the Oracle Key Vault server through SSH as user support, switch user su to root and then switch user su to oracle.
    3. Stop the agent by executing the following command:

       agent_installation_directory/bin/agentctl stop

    4. Log in to the Oracle Audit Vault Server console as an Oracle Audit Vault administrator.
    5. Delete the corresponding agent and target.
    6. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    7. Delete the installation directory for the Oracle Audit Vault agent.
14. Ensure that the Oracle Key Vault server certificate has not expired, nor is close to expiry, before you begin the upgrade.
    You can find how much time the Oracle Key Vault server certificate has before it expires by checking the OKV Server Certificate Expiration setting on the Configure Alerts page in the Oracle Key Vault management console.
15. If you are performing an upgrade while using an HSM as a Root of Trust, then consult Oracle Key Vault Root of Trust HSM Configuration Guide for any additional steps that may be needed.
16. Ensure that the backup of the orapwdbfwdb file matches the original file.
    1. SSH into the Oracle Key Vault system as user support, then switch to user root:

       ssh support@<OKV_IP_Address>
       su – root
       

    2. Verify that the backup file exists:

       su - oracle
       ls -ltr /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

    3. If the backup file exists, then perform the following steps:
       • Compare the original file with the backup file:

         diff /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

       • If there is a difference between the files, then update the backup file by copying the original file:

         cp /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

To upgrade the multi-master cluster, you must upgrade each multi-master cluster node, one after the other.

1. Ensure that you have performed the pre-upgrade steps.
2. Disable the multi-master cluster node.
   1. Log into the cluster node that you want to upgrade as a user with the System Administrator role.
   2. Select the Cluster tab, and then select Management from the left navigation bar.
   3. Under Cluster Details, in the Select Node column, select the check box of the node to disable.
   4. Click Disable.

      In the node’s Management page (under the Cluster tab), the node's status will change from DISABLING to DISABLED.

3. Ensure that SSH access is enabled for the node.
   Select the System tab, then Settings. In the Network Details area, click SSH Access. Select IP address(es) and then enter only the IP addresses that you need, or select All. Click Save.
4. Ensure that you have enough space in the destination directory for the upgrade ISO files.
5. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.

   ssh support@okv_server_IP_address
   su - root

   If the SSH connection times out while you are executing any step of the upgrade, then the operation will not complete successfully. Oracle recommends that you ensure that you use the appropriate values for the ServerAliveInterval and ServerAliveCountMax options for your SSH sessions to avoid upgrade failures.

   Using the tmux command prevents network disconnections interrupting the upgrade. If the session terminates, resume as follows:

   tmux a

6. Copy the upgrade ISO file to the destination directory using SCP or other secure transmission method.

   scp remote_host:remote_path/okv-upgrade-disc-new_software_release.iso /var/lib/oracle

   In this specification:

   • remote_host is the IP address of the computer containing the ISO upgrade file.
   • remote_path is the directory of the ISO upgrade file. Do not copy this file to any location other than the /var/lib/oracle directory.
7. Make the upgrade accessible by using the mount command:

   /bin/mount -o loop,ro /var/lib/oracle/okv-upgrade-disc-new_software_release.iso /images

8. Clear the cache using the clean all command:

   yum -c /images/upgrade.repo clean all

9. Apply the upgrade with the upgrade.rb command:

   root# ruby /images/upgrade.rb --confirm

   If the system is successfully upgraded, then the command will display the following message:

   Reboot now to continue the upgrade process.

   If you see an error message, then check the log file /var/log/messages for additional information.

   If the upgrade of the Oracle Key Vault system fails with the following message:

   Failed to apply update: The Oracle Key Vault upgrade has detected issues with FIPS mode. Please consult the Oracle Key Vault upgrade documentation or contact Oracle Support.

   Perform the following steps:

   1. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.

      ssh support@<Oracle_Key_Vault_IP_address> 
          su - root
      

   2. Run the following command:

      /images/preupgrade/okv_check_fips_status_utility fix_fips_mode_consistency

   3. Follow the instructions displayed in the output and reboot the system when prompted.
   4. After the system has successfully rebooted, SSH into the system again. As user root, mount the upgrade ISO and run the following command to verify that there is no FIPS mode inconsistency on the system:

      /images/preupgrade/okv_check_fips_status_utility check_for_fips_mode_consistency

      The return value 0 indicates that there is no more FIPS inconsistency.

      The return value 1 indicates that there is FIPS mode inconsistency. Run the following command to correct it:

      /images/preupgrade/okv_check_fips_status_utility fix_fips_mode_consistency

10. Restart the Oracle Key Vault server by running the reboot command:

    # reboot

    On the first restart of the computer after the upgrade, the system will apply the necessary changes. This can take a few hours. Do not shut down the system during this time. The upgrade of the cluster node is completed when the screen with heading: Oracle Key Vault Server new_software_release appears, with new_software_release reflecting the release number of the upgraded version. Following the heading appears the menu item Display Appliance Info. Select Display Appliance Info and press the Enter key to see the IP address settings for the appliance.

11. If you are performing an HSM upgrade using Entrust (formerly nCipher), then perform the additional steps described in Oracle Key Vault Root of Trust HSM Configuration Guide.
12. After the node has been successfully upgraded, re-enable it.
    1. Log into the Oracle Key Vault node that you just upgraded as a user who has the System Administrator role.
    2. Select the Cluster tab, and then select Management from the left navigation bar.
    3. In the Cluster Details section, under Name, click the name of the node that you had disabled.
    4. Click Enable.
       After you re-enable the disabled multi-master cluster node, its status changes from DISABLED to ENABLING, then to ACTIVE. The status of the node will remain at ENABLING and will not change to ACTIVE unless bidirectional replication between it and all other nodes is occurring successfully.
13. As necessary, disable SSH access on this node.
    Select the System tab, then Settings. In the Network Details area, click SSH Access. Select Disabled. Click Save.
14. After you have successfully completed this procedure, repeat these upgrade steps on all multi-master cluster nodes.

Nodes that were created in Oracle Key Vault releases earlier than release 21.1 use Classic mode, in which only one network interface was used.

If you prefer to use dual NIC network mode, which supports the use two network interfaces, then you can switch the node to use this mode, from the command line.

After you complete the upgrade of at least one node, you can log into any of the upgraded nodes to check the node and cluster versions.

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the Cluster tab.
3. In the left navigation bar, select Management.
4. Check the following areas:
   • To find the node version, check the Cluster Details area.
   • To find the cluster version, check the Cluster Information area.

If necessary, extend the swap space on each node. Oracle Key Vault release 21.11 requires a hard disk size greater than or equal to 1 TB in size with approximately 64 GB of swap space.

1. Log in to the server in which you upgraded Oracle Key Vault and connect as root.
2. Check the current amount of swap space.

   [root@my_okv_server support]# swapon -s

   Output similar to the following appears. This example shows that the system has 4 GB of swap space.

   Filename Type Size Used Priority
   /dev/dm-0 partition 4194300 3368 -1
   

   There must be 64 GB of swap space if the disk is greater than 1 TB in size.

3. Run the vgs command to determine how much free space is available.

   vgs

   The VFree column shows how much free space you have (for example, 21 GB).

4. Power off the server in order to add a new disk.

   /sbin/shutdown -h now

5. Add a new disk to the server of a size that will bring the VFree value to over 64 GB.
6. Start the server.
7. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.

   ssh support@okv_server_IP_address
   su - root
   

8. Run the fdisk -l command to find if there are any available partitions on the new disk.

   fdisk -l

   At this stage, there should be no available partitions.

9. Run the fdisk disk_device_to_be_added command to create the new partition.
   For example, to create a disk device named /dev/sdc:

   fdisk /dev/sdc

   In the prompts that appear, enter the following commands in sequence:

   • n for new partition
   • p for primary
   • 1 for partition number
   • Accept the default values for cylinder (press Enter twice).
   • w to write and exit
10. Use the pvcreate disk_device_partition command to add the newly added disk to the physical volume.
    For example, for a disk device named /dev/sdc1, which is the name of the disk partition that you created (based on the name used for the disk device that was added).

    pvcreate /dev/sdc1

    Output similar to the following appears:

    Physical volume "/dev/sdc1" successfully created

11. Use the vgextend vg_root disk_device_partition command to extend the logical volume with this disk space that you just added.
    For example, for the partition /dev/sdc1, you would run:

    vgextend vg_root /dev/sdc1

    Output similar to the following appears:

    Volume group "vg_root" successfully extended

12. Run the vgs command again to ensure that VFree shows an increase of 64 GB.

    vgs

13. Disable swapping.

    [root@my_okv_server support]# swapoff -v /dev/vg_root/lv_swap

14. To extend the swap space, run the lvresize command.

    [root@my_okv_server support]# lvresize -L +60G /dev/vg_root/lv_swap

    Output similar to the following appears:

    Size of logical volume vg_root/lv_swap changed from 4.00 GiB (128 extents) to 64.00 GiB (2048 extents)
    Logical volume lv_swap successfully resized.
    

15. Format the newly added swap space.

    [root@my_okv_server support]# mkswap /dev/vg_root/lv_swap

    Output similar to the following appears:

    mkswap: /dev/vg_root/lv_swap: warning: don't erase bootbits sectors
    on whole disk. Use -f to force.
    Setting up swapspace version 1, size = 67108860 KiB
    no label, UUID=fea7fc72-0fea-43a3-8e5d-e29955d46891
    

16. Enable swapping again.

    [root@my_okv_server support]# swapon -v /dev/vg_root/lv_swap

17. Verify the amount of swap space that is available.

    [root@my_okv_server support]# swapon -s

    Output similar to the following appears:

    Filename Type Size Used Priority 
    /dev/dm-0 partition 67108860 0 -1

18. Restart the Oracle Key Vault server.

    /sbin/reboot

    For primary-standby deployments, ensure that the primary and standby nodes sync up before proceeding further with next steps.

For each multi-master cluster node, Oracle recommends that you clean up the older kernels that were left behind after the upgrade.

1. Log in to the Oracle Key Vault server as the support user.
2. Switch to the root user.

   su - root

3. Mount /boot if it was not mounted on the system.
   1. Check if the /boot is mounted. The following command should display /boot information if it was mounted.

      df -h /boot;

   2. Mount it if /boot is not mounted.

      /bin/mount /boot;

4. Check the installed kernels and the running kernel.
   1. Search for any kernels that are installed.

      rpm -q kernel-uek | sort

      The following example output shows that two kernels are installed:

      kernel-uek-4.14.35-2047.504.2.el7uek.x86_64 
      kernel-uek-5.4.17-2136.329.3.1.el8uek.x86_64

   2. Check the latest kernel.

      uname -r

      The following output shows an example of a kernel version that was installed at the time:

      5.4.17-2136.304.4.5.el7uek.x86_64

      This example assumes 5.4.17-2136.304.4.5.el7uek.x86_64 as the latest version (newer versions may be available by now). Based on the output from the commands above, remove the older kernel (kernel-uek-4.14.35-2047.504.2.el7uek.x86_64). You should remove all kernels that are older than the latest kernel.

5. Remove the older kernel and its associated RPMs.

   For example, to remove the kernel-uek-4.14.35-2047.504.2.el7uek.x86_64 kernel:

   yum --disablerepo=* remove `rpm -qa|grep 4.14.35-2047.504.2.el7uek`

   Output similar to the following appears:

     Resolving Dependencies
   -->   Running transaction check
   ---> Package kernel-uek.x86_64 0:4.14.35-2047.504.2.el7uek will be erased
   ---> Package kernel-uek-devel.x86_64 0:4.14.35-2047.504.2.el7uek will be erased
   --> Finished Dependency Resolution
   
   Dependencies Resolved
   
   =================================================================================================================
    Package               Arch        Version                           Repository                          Size
   =================================================================================================================
   Removing:
    kernel-uek          x86_64         4.14.35-2047.504.2.el7uek       @anaconda/7.7                        58 M
    kernel-uek-devel    x86_64         4.14.35-2047.504.2.el7uek       @avs-ol-dependencies                 63 M
   
   Transaction Summary
   =================================================================================================================
   Remove        2 Package(s)
   
   Installed size: 121 M
   Is this ok [y/N]:

6. Enter y to accept the deletion output.
7. Repeat these steps starting with Step 4 for all kernels that are older than the latest kernel.

For each multi-master cluster node, you should remove SSH-related DSA keys left behind after the upgrade.

1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
2. Enable SSH.

   Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select IP address(es) and then enter only the IP addresses that you need, or select All. Click Save.

3. Log in to the Oracle Key Vault support account using SSH as the support user and then switch to the root user.

   ssh support@OracleKeyVault_serverIPaddress

   su - root

4. Change directory to /etc/ssh.

   cd /etc/ssh

5. Rename the following keys.

   mv ssh_host_dsa_key.pub ssh_host_dsa_key.pub.retire
   mv ssh_host_dsa_key ssh_host_dsa_key.retire

6. Disable SSH access.

   Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select Disabled. Click Save.