6 Upgrading a Standalone Oracle Key Vault Server

This upgrade includes the Oracle Key Vault server software and utilities that control the associated endpoint software

6.1 About Upgrading a Standalone Oracle Key Vault Server

To benefit from new features and security enhancements, Oracle recommends that you upgrade Oracle Key Vault server to the latest release.

You must upgrade in the following order: first perform a full backup of Oracle Key Vault, upgrade the Oracle Key Vault server, upgrade the endpoint software, and lastly, perform another full backup of the upgraded server. Note that upgrading requires a restart of the Oracle Key Vault server.

Oracle recommends using a multi-master cluster deployment for production use. During upgrade of a multi-master cluster, there is no downtime of databases or business applications. A two-node cluster provides read-only availability, and four or more node clusters provide continuous read-write availability. You can enable the persistent cache feature to enable endpoints to continue operation during the upgrade process.

When you upgrade the Oracle Key Vault server software, to access the latest enhancements, also upgrade the endpoint software. While endpoint software from the previous Oracle Key Vault release will continue to function with the upgraded Oracle Key Vault server, new endpoint functionality may not work.

Before you begin the upgrade, refer to Oracle Key Vault Release Notes for additional information about performing upgrades.

Related Topics

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.2 Step 1: Back Up the Server Before You Upgrade

Before you upgrade the Oracle Key Vault server, perform a one-time backup to a remote destination so that you can recover data in case the upgrade fails.

Caution:

Do not bypass this step. Back up the server before you perform the upgrade so that your data is safe and recoverable.

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.3 Step 2: Perform Pre-Upgrade Tasks for the Standalone Oracle Key Vault

To ensure a smooth upgrade to Oracle Key Vault, you should prepare the server you are upgrading.

  1. In the server where Oracle Key Vault is installed, log in as user support, and then switch to the root user.
  2. Ensure that the server meets the minimum disk space requirements for an upgrade. For example, 6 GB of free space in the /usr/local/dbfw/tmp directory. See the Oracle Key Vault Readme for this release to determine the disk space requirements for the upgrade.
  3. Ensure that you disable diagnostics and clean up disk space in /usr/local/dbfw/tmp before you upgrade by performing the following steps:

    If the Oracle Key Vault system being upgraded is from release 21.6 or later, log in to the Oracle Key Vault management console as a user with the System Administrator role, and navigate to the System tab, and then click the Diagnostics button.

    If the Diagnostics Package Files pane is displayed, then click Clear to disable diagnostics. Note that the Diagnostics Package Files pane will be displayed only if the diagnostics bundle was previously generated, and the files were not cleared.

    If the Diagnostics Package Files pane is not displayed, or if the diagnostics bundle was previously generated using the dbfw-diagnostics-package.rb utility, then log in to the Oracle Key Vault system and run the following commands to disable diagnostics and clean up disk space in /usr/local/dbfw/tmp:

    1. SSH into the Oracle Key Vault system as user support, then switch to user root:
      ssh support@<OKV_IP_Address>
su – root
    2. Delete the generated diagnostics zip file and remove the package using the following commands:
      /usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --clean
/usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --remove
  4. Check the boot partition size. If any of the nodes in question have a boot partition that is less than 500 MB, then you cannot upgrade that system to the new release. You can check this size as follows:
    1. Mount the /boot partition.
      # mount /boot
    2. Check the Size column given by the following command:
      # df -h /boot
    3. Unmount the /boot partition:
      # umount /boot
    If the boot partition given by this command shows less than 488 MB, then you cannot upgrade to the current release. Oracle recommends that you restore a backup of the current configuration to a freshly installed system of the same release as the current system, and upgrade that to the new release instead.
  5. If Oracle Key Vault is using the BIOS boot mode, then ensure that the disk size is not greater than 2 TB. If this is the case, then you cannot upgrade to the current release. Oracle recommends that you restore a backup of the current configuration onto a system with a disk that is less than 2 TB in size, and upgrade that to the new release instead.
  6. If you need to increase available disk space, then remove the temporary jar files located in /usr/local/okv/ssl. Be careful in doing so. If you accidentally delete any files other than the jar files in /usr/local/okv/ssl, then the Oracle Key Vault server becomes non-functional.
  7. Increase your disk space by extending the vg_root size:
    You must increase the disk space by extending vg_root before you perform the upgrade.
  8. Ensure that no full or incremental backup jobs are running. Delete all scheduled full or incremental backup jobs before the upgrade.
  9. Plan for downtime according to the following specifications:
    Oracle Key Vault Usage Downtime required

    Wallet upload or download

    NO

    Java Keystore upload or download

    NO

    Transparent Data Encryption (TDE) direct connect

    YES (NO with persistent cache)

    Primary Server Upgrade in a primary-standby deployment

    YES (NO with persistent cache)
  10. Plan for downtimes.
    If Oracle Key Vault uses an online master encryption key, then plan for a downtime of 15 minutes during the Oracle Database endpoint software upgrades. Database endpoints can be upgraded in parallel to reduce total downtime.
  11. If the Oracle Key Vault system has a syslog destination configured, ensure that the remote syslog destination is reachable from the Oracle Key Vault system, and that logs are being correctly forwarded. If the remote syslog destination is not reachable from the Oracle Key Vault system, then the upgrade process can become much slower than normal.
  12. If Oracle Audit Vault was integrated with Oracle Key Vault release 21.2 or earlier, then do the following to disable and remove the Oracle Audit Vault integration:
    1. Disable the Oracle Audit Vault integration: Log into the Oracle Key Vault management console as a System Administrator, select the System tab and then Settings from the left navigation bar. In the Monitoring and Alerts pane, select Audit Vault. In the Audit Vault integration pane that appears, disable AVDF. Click Save.
    2. Log in to the Oracle Key Vault server through SSH as user support, switch user su to root and then switch user su to oracle.
    3. Stop the agent by executing the following command:
      agent_installation_directory/bin/agentctl stop
    4. Log in to the Oracle Audit Vault Server console as an Oracle Audit Vault administrator.
    5. Delete the corresponding agent and target.
    6. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    7. Delete the installation directory for the Oracle Audit Vault agent.
  13. If you are performing an upgrade while using an HSM as a Root of Trust, then consult Oracle Key Vault Root of Trust HSM Configuration Guide for any additional steps that may be needed.
  14. Ensure that the Oracle Key Vault server certificate has not expired, nor is close to expiry, before you begin the upgrade.
    You can find how much time the Oracle Key Vault server certificate has before it expires by checking the OKV Server Certificate Expiration setting on the Configure Alerts page in the Oracle Key Vault management console.
  15. Ensure that the backup of the orapwdbfwdb file matches the original file.
    1. SSH into the Oracle Key Vault system as user support, then switch to user root:
      ssh support@<OKV_IP_Address>
su – root
    2. Verify that the backup file exists:
      su - oracle
ls -ltr /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb
    3. If the backup file exists, then perform the following steps:
      • Compare the original file with the backup file:
        diff /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb
      • If there is a difference between the files, then update the backup file by copying the original file:
        cp /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

Related Topics

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.4 Step 3: Add Disk Space to Extend the vg_root for the Release 21.11 Upgrade

Before upgrading from Oracle Key Vault release 12.2 or 18 to 21, you need to extend the vg_root to increase disk space.

If you are upgrading from an earlier Oracle Key Vault release 21.x and have already extended the vg_root, then you can bypass this step.
Before you start this procedure, ensure that all endpoints have persistent cache enabled and in use.
  1. Log in to the server for which you will perform the upgrade and switch user as root.
  2. Ensure that the persistent cache settings for Oracle Key Vault have been set.
    You will need to ensure that the persistent cache has been enabled because in a later step in this procedure, you must shut down the server. Shutting down the Oracle Key Vault server will incur downtime. To avoid any downtime, Oracle recommends that you turn on persistent cache.
  3. Run the vgs command to determine the free space.
    vgs

    The VFree column shows how much free space you have (for example, 21 GB).

  4. Power off the server in order to add a new disk.
    /sbin/shutdown -h now
  5. Add a new disk to the server with a capacity of 100 GB or greater.
  6. Start the server.
  7. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    ssh support@okv_server_IP_address
su - root
  8. Stop the Oracle Key Vault services.
    service tomcat stop;
service httpd stop;
service kmipus stop;
service kmip stop;
service okvogg stop;
service javafwk stop;
service monitor stop;
service controller stop;
service dbfwlistener stop;
service dbfwdb stop;
service rsyslog stop;
  9. Run the fdisk -l command to find if there are any available partitions on the new disk.
    fdisk -l
    At this stage, there should be no available partitions.
  10. Run the fdisk disk_device_to_be_added command to create the new partition.
    For example, to create a disk device named /dev/sdb:
    fdisk /dev/sdb

    In the prompts that appear, enter the following commands in sequence:

    • n for new partition
    • p for primary
    • 1 for partition number
    • Accept the default values for cylinder (press Enter twice).
    • w to write and exit
  11. Use the pvcreate disk_device_partition command to add the newly added disk to the physical volume.
    For example, for a disk device named /dev/sdb1, which is the name of the disk partition that you created (based on the name used for the disk device that was added).
    pvcreate /dev/sdb1

    Output similar to the following appears:

    Physical volume "/dev/sdb1" successfully created
  12. Use the vgextend vg_root disk_device_partition command to extend the logical volume with this disk space that you just added.
    For example, for the partition /dev/sdb1, you would run:
    vgextend vg_root /dev/sdb1

    Output similar to the following appears:

    Volume group "vg_root" successfully extended
  13. Run the vgs command again to ensure that VFree shows an increase of 100 GB or more (depending on the size of the disk that was added).
    vgs

    Output similar to the following appears:

    VG      #PV #LV #SN Attr   VSize   VFree
vg_root   2  12   0 wz--n- 598.75g <121.41g
  14. Restart the Oracle Key Vault server.
    /sbin/reboot

Related Topics

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.5 Step 4: Upgrade the Oracle Key Vault Server

You can upgrade a standalone Oracle Key Vault server deployment.

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.5.1 About Upgrading an Oracle Key Vault Server

In a standalone deployment you must upgrade a single Oracle Key Vault server.

Note that persistent caching enables endpoints to continue to be operational during the upgrade process.

Note:

If you are upgrading from a system with 4 GB RAM, first add 12 GB or more of additional RAM, following instructions for your specific hardware, before upgrading. Ensure that the persistent cache is enabled and set to sufficiently large values before attempting such operations so as to not incur endpoint downtime.

Related Topics

Parent topic: Step 4: Upgrade the Oracle Key Vault Server

6.5.2 Upgrading a Standalone Oracle Key Vault Server

A single Oracle Key Vault server in a standalone deployment is sometimes used in test and development environments for functional testing.

  1. Ensure that you have backed up the server you are upgrading so your data is safe and recoverable.
    Do not proceed without completing this step.
  2. Log into the Oracle Key Vault management console as a user who has the System Administrator role.
  3. Ensure that SSH access is enabled.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select IP address(es) and then enter only the IP addresses that you need, or select All. Click Save.

  4. Ensure you have enough space in the destination directory for the upgrade ISO files.
    Do not copy this file to any location other than the /var/lib/oracle directory.
  5. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    ssh support@okv_server_IP_address
su - root
    If the SSH connection times out while you are executing any step of the upgrade, then the operation will not complete successfully. Oracle recommends that you ensure that you use the appropriate values for the ServerAliveInterval and ServerAliveCountMax options for your SSH sessions to avoid upgrade failures.Using the tmux command prevents network disconnections interrupting the upgrade. If the session terminates, resume as follows:
    root# tmux a
  6. Copy the upgrade ISO file to the destination directory using Secure Copy Protocol or other secure transmission method.

    Note:

    The upgrade ISO file is not the installation ISO file that you downloaded from eDelivery. You can download the Oracle Key Vault 21.11 upgrade software from https://updates.oracle.com/download/37484096.html. 
    root# scp user_name@remote_host:remote_path/okv-upgrade-21.11.0.0.0.iso /var/lib/oracle/

    In this specification:

    • remote_host is the IP address of the computer containing the ISO upgrade file.
    • remote_path is the directory of the ISO upgrade file. Do not copy this file to any location other than the /var/lib/oracle directory.
  7. As root, make the upgrade accessible by using the mount command:
    root# mount -o loop,ro /var/lib/oracle/okv-upgrade-21.11.0.0.0.iso /images
  8. Clear the cache using the clean all command:
    root# yum -c /images/upgrade.repo clean all
  9. Apply the upgrade with the upgrade.rb command:
    root# ruby /images/upgrade.rb --confirm

    If the system is successfully upgraded, then the command will display the following message:

    Reboot now to continue the upgrade process.

    If you see an error message, then check the log file /var/log/messages for additional information.

    If the upgrade of the Oracle Key Vault system fails with the following message:

    Failed to apply update: The Oracle Key Vault upgrade has detected issues with FIPS mode. Please consult the Oracle Key Vault upgrade documentation or contact Oracle Support.

    Perform the following steps:

    1. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
      ssh support@<Oracle_Key_Vault_IP_address> 
    su - root
    2. Run the following command:
      /images/preupgrade/okv_check_fips_status_utility fix_fips_mode_consistency
    3. Follow the instructions displayed in the output and reboot the system when prompted.
    4. After the system has successfully rebooted, SSH into the system again. As user root, mount the upgrade ISO and run the following command to verify that there is no FIPS mode inconsistency on the system:
      /images/preupgrade/okv_check_fips_status_utility check_for_fips_mode_consistency

      The return value 0 indicates that there is no more FIPS inconsistency.

      The return value 1 indicates that there is FIPS mode inconsistency. Run the following command to correct it:

      /images/preupgrade/okv_check_fips_status_utility fix_fips_mode_consistency
  10. Restart the Oracle Key Vault server by running the reboot command:
    # reboot

    On the first restart of the computer after the upgrade, assuming that the upgrade ISO file was copied to the /var/lib/oracle directory, the system will automatically mount /var/lib/oracle/okv-upgrade-21.11.0.0.0.iso and finish the upgrade process. (If the ISO is not auto-mounted, then the upgrade process will prompt for the ISO to be re-attached.) This can take a few hours. Do not shut down the system during this time.

    The upgrade is complete when the screen shows the following text: Oracle Key Vault Server version. This appliance was upgraded from previous_release_version. The revision reflects the upgraded release.

  11. Confirm that Oracle Key Vault has been upgraded to the correct version.
    1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    2. Select the System tab, and then select Status.
    3. Verify that the version displayed is the latest release number.
      The release number is also at the bottom of each page, to the right of the copyright information.
  12. If your site uses the Commercial National Security Algorithm (CNSA) suite, then re-install these algorithms onto the standalone server.
  13. Restart the Oracle Key Vault system.
    root# /sbin/reboot
  14. Delete the upgrade ISO from the Oracle Key Vault server that was just upgraded.
    For example:
    root# /bin/rm -f /var/lib/oracle/okv-upgrade-21.11.0.0.0.iso
  15. Disable SSH access.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select Disabled. Click Save.

Related Topics

Parent topic: Step 4: Upgrade the Oracle Key Vault Server

6.5.2.1 Correct System Inconsistencies Before Upgrade

You can correct the system inconsistencies before upgrading to the latest Oracle Key Vault release.

If your upgrade path includes Oracle Key Vault release 21.1 or 21.2 and while running the system at Oracle Key Vault release 21.1 or 21.2 you have enabled or disabled FIPS mode, the upgrade to Oracle Key Vault 21.11 may result in an error. The error is because all components in Oracle Key Vault do not have the same mode when it comes to FIPS, that is, they are not all enabled or all disabled. All the system components should work with the similar FIPS mode, that is, all the components should be with FIPS mode enabled or disabled before you can proceed with upgrade.

You get the following error on upgrade if FIPS mode is not consistent in Oracle Key Vault,

 # ruby /images/upgrade.rb --confirm

Power loss during upgrade may cause data loss. Do not power
off during upgrade.
Verifying boot partition before upgrade 
Failed to apply update: 
The Oracle Key Vault upgrade has detected issues with FIPS mode. 
Please consult the Oracle Key Vault upgrade documentation or contact Oracle Support.

Before you upgrade, follow the steps to fix the inconsistent state of FIPS.

  1. SSH into the Oracle Key Vault system as user support, then switch to user root.
    ssh support@<Oracle_Key_Vault_IP_address> 
su - root
  2. Run the following commands:
    su - oracle -c "/usr/local/okv/bin/fips_nzzt_enable"
su - oracle -c "/usr/local/okv/bin/fips_ogg_enable"
FIPS_ENABLED in /usr/local/okv/etc/okv_security.conf updated from "0" to "1":
sed -i "/^FIPS_ENABLED=/cFIPS_ENABLED=\"1\"" /usr/local/okv/etc/okv_security.conf
  3. Reboot the system.
  4. Upgrade the system after the reboot is complete.

6.6 Step 5: If Necessary, Add Disk Space to Extend Swap Space

If necessary, extend the swap space. Oracle Key Vault release 21.11 requires a hard disk size greater than or equal to 2 TB in size with approximately 64 GB of swap space.

If your system does not meet this requirement, follow these instructions to extend the swap space. You can check how much swap space you have by running the swapon -s command. By default, Oracle Key Vault releases earlier than release 18.1 were installed with approximately 4 GB of swap space. After you complete the upgrade to release 18.1 or later, Oracle recommends that you increase the swap space allocation for the server on which you upgraded Oracle Key Vault. A new Oracle Key Vault installation is automatically configured with sufficient swap space. However, if you upgraded from a previous release, and your system does not have the desired amount of swap space configured, then you must manually add disk space to extend the swap space, particularly if the intention is to convert the upgraded server into the first node of a multi-master cluster.
  1. Log in to the server in which you upgraded Oracle Key Vault and connect as root.
  2. Check the current amount of swap space.
    [root@my_okv_server support]# swapon -s

    Output similar to the following appears. This example shows that the system has 4 GB of swap space.

    Filename Type Size Used Priority
/dev/dm-0 partition 4194300 3368 -1

    There must be 64 GB of swap space if the disk is greater than 1 TB in size.

  3. Run the vgs command to determine how much free space is available.
    vgs

    The VFree column shows how much free space you have (for example, 21 GB).

  4. Power off the server in order to add a new disk.
    /sbin/shutdown -h now
  5. Add a new disk to the server of a size that will bring the VFree value to over 64 GB.
  6. Start the server.
  7. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    ssh support@okv_server_IP_address
su - root
  8. Run the fdisk -l command to find if there are any available partitions on the new disk.
    fdisk -l

    At this stage, there should be no available partitions.

  9. Run the fdisk disk_device_to_be_added command to create the new partition.
    For example, to create a disk device named /dev/sdc:
    fdisk /dev/sdc

    In the prompts that appear, enter the following commands in sequence:

    • n for new partition
    • p for primary (the primary partition)
    • 1 for partition number
    • Accept the default values for cylinder (press Enter twice).
    • w to write and exit
  10. Use the pvcreate disk_device_partition command to add the newly added disk to the physical volume.
    For example, for a disk device named /dev/sdc1, which is the name of the disk partition that you created (based on the name used for the disk device that was added).
    pvcreate /dev/sdc1

    Output similar to the following appears:

    Physical volume "/dev/sdc1" successfully created
  11. Use the vgextend vg_root disk_device_partition command to extend the logical volume with this disk space that you just added.
    For example, for the partition /dev/sdc1, you would run:
    vgextend vg_root /dev/sdc1

    Output similar to the following appears:

    Volume group "vg_root" successfully extended
  12. Run the vgs command again to ensure that VFree shows an increase of 64 GB.
    vgs
  13. Disable swapping.
    [root@my_okv_server support]# swapoff -v /dev/vg_root/lv_swap
  14. To extend the swap space, run the lvresize command.
    [root@my_okv_server support]# lvresize -L +60G /dev/vg_root/lv_swap

    Output similar to the following appears: 

    Size of logical volume vg_root/lv_swap changed from 4.00 GiB (128 extents) to 64.00 GiB (2048 extents)
Logical volume lv_swap successfully resized.
  15. Format the newly added swap space.
    [root@my_okv_server support]# mkswap /dev/vg_root/lv_swap

    Output similar to the following appears:

    mkswap: /dev/vg_root/lv_swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 67108860 KiB
no label, UUID=fea7fc72-0fea-43a3-8e5d-e29955d46891
  16. Enable swapping again.
    [root@my_okv_server support]# swapon -v /dev/vg_root/lv_swap
  17. Verify the amount of swap space that is available.
    [root@my_okv_server support]# swapon -s

    Output similar to the following appears:

    Filename Type Size Used Priority 
/dev/dm-0 partition 67108860 0 -1
  18. Restart the Oracle Key Vault server.
    /sbin/reboot

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.7 Step 6: If Necessary, Remove Old Kernels

Oracle recommends that you clean up the older kernels that were left behind after the upgrade.

While the older kernel is not in use, it may be marked as an issue by some code analysis tools.
  1. Log in to the Oracle Key Vault server as the support user.
  2. Switch to the root user.
    su - root
  3. Mount /boot if it was not mounted on the system.
    1. Check if the /boot is mounted. The following command should display /boot information if it was mounted.
      df -h /boot;
    2. Mount it if /boot is not mounted.
      /bin/mount /boot;

      For EFI-based systems, you may need to mount /boot/efi if it is not already mounted. 

      /bin/mount /boot/efi
  4. Check the installed kernels and the running kernel.
    1. Search for any kernels that are installed. 
      rpm -q kernel-uek | sort;

      The following example output shows that two kernels are installed:

      kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64
kernel-uek-5.4.17-2136.329.3.1.el8uek.x86_64
    2. Check the latest kernel. 
      uname -r;

      The following output shows an example of a kernel version that was installed at the time:

      5.4.17-2136.329.3.1.el8uek.x86_64

      This example assumes 5.4.17-2136.329.3.1.el8uek.x86_64 as the latest version (newer versions may be available by now). Based on the output from the commands above, remove the older kernel (kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64). You should remove all kernels that are older than the latest kernel.

  5. Remove the older kernel and its associated RPMs.

    For example, to remove the kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64

    # yum --disablerepo=* remove `rpm -qa|grep kernel-uek-5.4.17-2136.318.7.2.el8uek`

    Output similar to the following appears: 

      Resolving Dependencies
-->   Running transaction check
---> Package kernel-uek.x86_64 0:4.14.35-2047.504.2.el7uek will be erased
---> Package kernel-uek-devel.x86_64 0:4.14.35-2047.504.2.el7uek will be erased
--> Finished Dependency Resolution
Dependencies resolved.
================================================================================
 Package Arch Version Repository Size
================================================================================
Removing:
 kernel-uek x86_64 5.4.17-2136.318.7.2.el8uek @avdf-base-os 135 M

Transaction Summary
================================================================================
Remove 1 Package

Freed space: 135 M
Is this ok [y/N]:
  6. Enter y to accept the deletion output.
  7. Repeat these steps starting with Step 4 for all kernels that are older than the latest kernel.

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.8 Step 7: If Necessary, Remove SSH-Related DSA Keys

You should remove SSH-related DSA keys left behind after the upgrade, because they can cause problems with some code analysis tools.

  1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
  2. Enable SSH.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select IP address(es) and then enter only the IP addresses that you need, or select All. Click Save.

  3. Log in to the Oracle Key Vault support account using SSH as the support user and then switch to the root user.
    ssh support@OracleKeyVault_serverIPaddress
    su - root
  4. Change directory to /etc/ssh.
    cd /etc/ssh
  5. Rename the following keys. 
    mv ssh_host_dsa_key.pub ssh_host_dsa_key.pub.retire
mv ssh_host_dsa_key ssh_host_dsa_key.retire
  6. Disable SSH access.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab, then Settings. In the Network Details area, click SSH Access. Select Disabled. Click Save.

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.9 Step 8: Upgrade the Endpoint Software

When you upgrade the Oracle Key Vault server software appliance, also upgrade the endpoint software to get access to the latest enhancements.

Oracle Key Vault client software is backward-compatible. While older versions of Oracle Key Vault client software are fully functional with an upgraded Oracle Key Vault server, some new Oracle Key Vault features are only available with the current client software.

You can upgrade an endpoint by upgrading the endpoint software or re-enrolling the endpoint. Upgrading the endpoint software does not affect the existing endpoint certificate or okvclient.ora, the endpoint configuration file. Re-enrolling an endpoint invalidates an existing endpoint certificate, and a new endpoint certificate as well as okvclient.ora are installed. Oracle recommends that you upgrade the endpoint software for minor version upgrades (for example, from 21.x to 21.y) and consider re-enrolling the endpoint when upgrading across major versions (for example, from 18.x to 21.y).

Before an endpoint that uses Oracle Key Vault for TDE key management can take advantage of new Oracle Key Vault features, for example non-extractable TDE master keys, it must be upgraded to match the new Oracle Key Vault release.

  1. For the endpoint upgrade of a TDE-enabled database, the database instance must be shut down to install the latest PKCS#11 library. Oracle recommends upgrading all endpoints for TDE-enabled databases on the same host together. Review the instructions in step 6 before proceeding with the upgrade of endpoints for TDE-enabled databases.
    You can upgrade an endpoint by updating the endpoint software or by re-enrolling the endpoint. Perform steps 2 - 4 to update the endpoint software.

    Or

    Perform step 5 to re-enroll the endpoint.

  2. Download the endpoint software (okvclient.jar) and install it in your existing endpoint directory path as follows:
    1. Go to the Oracle Key Vault management console login screen.
    2. Click the Endpoint Enrollment and Software Download link.
    3. In the Download Endpoint Software Only section, select the appropriate platform from the drop-down list.
    4. Click the Download button to download the okvclient.jar file.
  3. Identify the path to your existing endpoint installation that you are about to upgrade. For example, /etc/ORACLE/KEYSTORES/okv (where /etc/ORACLE/KEYSTORES is WALLET_ROOT of your database, or the softlink in $ORACLE_BASE/okv/$ORACLE_SID points to).
  4. Install the endpoint software by running the following command:
    java -jar okvclient.jar -d existing_endpoint_directory_path
    For example:
    java -jar okvclient.jar -d /etc/ORACLE/KEYSTORES/okv

    If you are installing the okvclient.jar file for an endpoint that has Oracle Database 23ai, then include the -arch db23ai option during the installation. The new endpoint software for Oracle Database 23ai is required to support new features such as using OpenSSL for FIPS mode and the new version of local auto login wallets in Oracle Database 23ai. The new endpoint software for Oracle Database 23ai is supported on the Linux-x64 platform only.

    For example:
    Java -jar okvclient.jar -d /home/oracle/okvutil -arch db23ai
  5. Perform the following steps to re-enroll the endpoint software, which also generates a new endpoint certificate. The easiest way to re-enroll an endpoint is by using the following commands of the RESTful services utility:
    1. Re-enroll the endpoint by using the following RESTful services utility command: 
      okv admin endpoint re-enroll
    2. Back up the OKV_HOME directory and delete the files under OKV_HOME: 
      cp -R $OKV_HOME $OKV_HOME_bkp_date +%Y%m%d
    3. Go to the $OKV_HOME directory and remove all the files.
    4. For Oracle Database 21c and earlier:
      Download and install the endpoint software by using the following RESTful services utility command:
      okv admin endpoint provision

      For Oracle Database 23ai:

      Download and install the endpoint software by using the following RESTful services utility command: 
      okv admin endpoint provision --arch db23ai

    Re-enrolling an endpoint generates a new okvclient.jar file and installs the file in the OKV_HOME directory but maintains the relationship between the endpoint and its default wallet.

    Note:

    To re-enroll an endpoint without using RESTful services utility, follow the steps described in How to Re-enroll an Endpoint.
  6. Install the updated PKCS#11 library file.
    This step is needed only for online TDE master encryption key management by Oracle Key Vault. If an endpoint uses online TDE master encryption key management by Oracle Key Vault, then you must upgrade the PKCS#11 library while upgrading the endpoint software.

    For Oracle Database 21c and earlier:

    Ensure that database instance is shut down before installing the PKCS#11 library in the location /opt/oracle/extapi/64/hsm/oracle/1.0.0.

    • On UNIX/Linux platforms: Run root.sh from the bin directory of endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints.
      $ sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh

      Or

      $ su - root
# /etc/ORACLE/KEYSTORES/okv/bin/root.sh
    • On Windows platforms: Run root.bat from the bin directory of endpoint installation directory to copy the latest liborapkcs.dll file for Oracle Database endpoints. You will be prompted for the version of the database in use.
      bin\root.bat

    If you are upgrading multiple endpoints for TDE-enabled databases on a host, you must install the latest Oracle Key Vault PKCS#11 library only once on the host computer.

    On the host, perform the following steps when upgrading multiple endpoints for TDE-enabled databases:

    • Complete the upgrade of all Oracle Key Vault endpoints for the TDE-enabled databases.
    • Shut down corresponding TDE-enabled database instances.
    • Execute the root.sh or root.bat script to install the latest Oracle Key Vault PKCS#11 library.

    For Oracle Database 23ai: Oracle recommends that you install the latest liborapkcs.so file in a fixed custom location using the root.sh script.

    This enables upgrading the liborapkcs.so file in future without encountering database downtime.

    The fixed custom location is in the following format:
     /opt/oracle/extapi/64/pkcs11/okv/<okv_version>/lib
    For example, in the Oracle Key Vault 21.11 release, the location is
     /opt/oracle/extapi/64/pkcs11/okv/21.11.0.0.0/lib
    • On UNIX/Linux platforms: Run root.sh from the bin directory of the endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints.
      $ sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh –-okv_pkcs11_library_location

      Or

      $ su - root
# /etc/ORACLE/KEYSTORES/okv/bin/root.sh –okv_pkcs11_library_location

    If you are upgrading multiple endpoints for TDE-enabled databases on a host, you must install the latest Oracle Key Vault PKCS#11 library only once for the endpoint software version.

    On the host where there are multiple endpoints for TDE-enabled Oracle database 23ai, each database can upgrade their endpoint software separately and switch to using the upgraded liborapkcs.so library by setting up the database initialization parameter PKCS11_LIBRARY_LOCATION to point to the upgraded library.

    Note:

    Installing the liborapkcs.so library in a legacy location is also supported with Oracle Database 23ai databases. However, Oracle does not recommend it.
  7. Update the SDK software.
    If you have already deployed the SDK software, Oracle recommends that you redeploy the SDK software in the same location after you complete the upgrade to Oracle Key Vault release 21.10. This enables you to have access to the new SDK APIs that were introduced since the Oracle Key Vault version that you are upgrading from.
    1. Go to the Oracle Key Vault management console login screen.
    2. Click the Endpoint Enrollment and Software Download link.
    3. c. In the Download Software Development Kit section, select the appropriate language and platform for your site.
    4. Click the Download button to get the SDK zip file.
    5. Identify the existing location where SDK software was already deployed.
    6. Navigate to the directory in which you saved the SDK zip file.
    7. Unzip the SDK zip file.
      For example, on Linux, to unzip the Java SDK zip file, use the following command:
      unzip -o okv_jsdk.zip -d existing_endpoint_sdk_directory_path
      For the C SDK zip file, use this command:
      unzip -o okv_csdk.zip -d existing_endpoint_sdk_directory_path
    8. Do not exit this page.
  8. If you had deployed the RESTful services utility in the previous release, then re-deploy the latest okvrestclipackage.zip file.

    The latest okvrestclipackage.zip file enables you to have access to the new RESTful services utility commands that were introduced since the Oracle Key Vault version that you are upgrading from.

    You can use wget or curl to download okvrestclipackage.zip. 

    wget --no-check-
certificate https://Oracle_Key_Vault_IP_address:5695/ okvrestclipackage.zip curl -O -
k https://Oracle_Key_Vault_IP_address:5695/okvrestservices.jar
  9. Start the Oracle Databases if the upgrade of Oracle Key Vault endpoints for all of the TDE enabled databases on this host machine is complete.

    At this stage, the endpoint is fully upgraded.

    For Oracle Database 23ai which is using the custom path for installing the liborapkcs.so library from step 6 above, perform the following additional steps.

    1. Log in to the CDB Root as a user who has been granted the ALTER SYSTEM privilege.
    2. Set the static initialization parameter PKCS11_LIBRARY_LOCATION to point to the upgraded liborapkcs.so library.
      For example:
      ALTER SYSTEM SET
PKCS11_LIBRARY_LOCATION=’/opt/oracle/extapi/64/pkcs11/okv/21.11.0.0.0/lib/liborapkcs.so’ SCOPE=SPFILE SID=’*’;

    After the database restart, the database will switch to using the new upgraded library from the custom path. The database is also set to switch to new libraries in future without encountering downtime.

    After you upgrade Oracle Key Vault to a new release in future (for example 21.xx.0.0.0), the root.sh file in the upgraded endpoint can be used to copy the liborapkcs.so library to the new path /opt/oracle/extapi/64/pkcs11/21.12.0.0.0/lib. The database can switch to the new library by using the SWITCHOVER command.

    For example:
    ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY 
‘/opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib/liborapkcs.so’ 
FOR ALL CONTAINERS;
  10. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Real Application Clusters (Oracle RAC) environment, then perform the following steps on each Oracle RAC node:
    1. Perform the endpoint upgrade on each Oracle RAC node.
    2. Set the extractable attribute value for symmetric keys.
      By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then you must set this value to false. You can set an extractable attribute value as follows:
      • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific setting overrides the global endpoint settings.
      • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
      • Modify the extractable attribute of an existing symmetric key.
    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the Oracle RAC node. Use the following syntax: 
      ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
[FORCE KEYSTORE][USING TAG 'tag_name'] 
IDENTIFIED BY [EXTERNAL STORE | keystore_password]
[WITH BACKUP [USING backup_identifier']];

      See Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.

  11. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Data Guard environment, then do the following on the primary and standby databases:
    1. Perform the endpoint upgrade on the primary and standby databases.
    2. Set the extractable attribute value for symmetric keys.
      By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then you must set this value to false. You can set an extractable attribute value as follows:
      • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific setting overrides the global endpoint settings.
      • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
      • Modify the extractable attribute of an existing symmetric key.
    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the primary and standby databases. Use the following syntax: 
      ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
[FORCE KEYSTORE]
[USING TAG 'tag_name'] 
IDENTIFIED BY [EXTERNAL STORE | keystore_password]
[WITH BACKUP [USING 'backup_identifier'']];

      See Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.

Related Topics

Parent topic: Upgrading a Standalone Oracle Key Vault Server

6.10 Step 9: Back Up the Upgraded Oracle Key Vault Server

You must perform server backup and user password tasks after completing a successful upgrade.

  1. Take a full backup of the upgraded Oracle Key Vault Server Database to a new remote destination. Avoid using the old backup destination for the new backups.
  2. Schedule a new periodic incremental backup to the new destination defined in the preceding step.
  3. Change the Oracle Key Vault administrative passwords.
    Password hashing has been upgraded to a more secure standard than in earlier releases. This change affects the operating system passwords, support and root. You must change Oracle Key Vault administrative passwords after the upgrade to take advantage of the more secure hash.

Parent topic: Upgrading a Standalone Oracle Key Vault Server