Before you upgrade the Oracle Key Vault server, perform a one-time backup to a remote destination so that you can recover data in case the upgrade fails.

To ensure a smooth upgrade to Oracle Key Vault, you should prepare the server you are upgrading.

1. In the server where Oracle Key Vault is installed, log in as user support, and then switch to the root user.
2. Ensure that the server meets the minimum disk space requirements for an upgrade. For example, 6 GB of free space in the /usr/local/dbfw/tmp directory. See the Oracle Key Vault Readme for this release to determine the disk space requirements for the upgrade.
3. Ensure that you disable diagnostics and clean up disk space in /usr/local/dbfw/tmp before you upgrade by performing the following steps:

   If the Oracle Key Vault system being upgraded is from release 21.6 or later, log in to the Oracle Key Vault management console as a user with the System Administrator role, and navigate to the System tab, and then click the Diagnostics button.

   If the Diagnostics Package Files pane is displayed, then click Clear to disable diagnostics. Note that the Diagnostics Package Files pane will be displayed only if the diagnostics bundle was previously generated, and the files were not cleared.

   If the Diagnostics Package Files pane is not displayed, or if the diagnostics bundle was previously generated using the dbfw-diagnostics-package.rb utility, then log in to the Oracle Key Vault system and run the following commands to disable diagnostics and clean up disk space in /usr/local/dbfw/tmp:

   1. SSH into the Oracle Key Vault system as user support, then switch to user root:

      ssh support@<OKV_IP_Address>
      su – root
      

   2. Delete the generated diagnostics zip file and remove the package using the following commands:

      /usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --clean
      /usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --remove

4. Check the boot partition size. If any of the nodes in question have a boot partition that is less than 500 MB, then you cannot upgrade that system to the new release. You can check this size as follows:
   1. Mount the /boot partition.

      # mount /boot

   2. Check the Size column given by the following command:

      # df -h /boot

   3. Unmount the /boot partition:

      # umount /boot

   If the boot partition given by this command shows less than 488 MB, then you cannot upgrade to the current release. Oracle recommends that you restore a backup of the current configuration to a freshly installed system of the same release as the current system, and upgrade that to the new release instead.
5. If Oracle Key Vault is using the BIOS boot mode, then ensure that the disk size is not greater than 2 TB. If this is the case, then you cannot upgrade to the current release. Oracle recommends that you restore a backup of the current configuration onto a system with a disk that is less than 2 TB in size, and upgrade that to the new release instead.
6. If you need to increase available disk space, then remove the temporary jar files located in /usr/local/okv/ssl. Be careful in doing so. If you accidentally delete any files other than the jar files in /usr/local/okv/ssl, then the Oracle Key Vault server becomes non-functional.
7. Increase your disk space by extending the vg_root size.
   You must increase the disk space by extending vg_root before you perform the upgrade.
8. Ensure that no full or incremental backup jobs are running. Delete all scheduled full or incremental backup jobs before the upgrade.
9. Plan for downtime according to the following specifications:

   Oracle Key Vault Usage	Downtime required
   Wallet upload or download	NO
   Java Keystore upload or download	NO
   Transparent Data Encryption (TDE) direct connect	YES (NO with persistent cache)
   Primary Server Upgrade in a primary-standby deployment	YES (NO with persistent cache)

10. Plan for downtimes.
    • If Oracle Key Vault uses an online master encryption key, then plan for a downtime of 15 minutes during the Oracle Database endpoint software upgrades. Database endpoints can be upgraded in parallel to reduce total downtime.
    • Plan for a downtime of a few hours. The persistent cache allows you to upgrade Oracle Key Vault servers without database downtime. The default duration of the persistent cache from the moment the Oracle Key Vault server becomes unavailable is 1,440 minutes (one day).
11. If the Oracle Key Vault system has a syslog destination configured, ensure that the remote syslog destination is reachable from the Oracle Key Vault system, and that logs are being correctly forwarded. If the remote syslog destination is not reachable from the Oracle Key Vault system, then the upgrade process can become much slower than normal.
12. If Oracle Audit Vault was integrated with Oracle Key Vault in Oracle Key Vault release 21.2 or earlier, then do the following on the primary server (but not the standby) to disable and remove the Oracle Audit Vault integration:
    1. Disable the Oracle Audit Vault integration: Log into the Oracle Key Vault management console as a System Administrator, select the System tab and then Settings from the left navigation bar. In the Monitoring and Alerts pane, select Audit Vault. In the Audit Vault integration pane that appears, disable Oracle Audit Vault. Click Save.
    2. Perform the remaining steps in this procedure on each server where the Oracle Audit Vault integration was configured.
    3. Log in to the Oracle Key Vault server through SSH as user support, switch user su to root and then switch user su to oracle.
    4. Stop the agent by executing the following command:

       agent_installation_directory/bin/agentctl stop

    5. Log in to the Oracle Audit Vault Server console as an Oracle Audit Vault administrator.
    6. Delete the corresponding agent and target.
    7. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.
    8. Delete the installation directory for the Oracle Audit Vault agent.
13. Ensure that the Oracle Key Vault server certificate has not expired, nor is close to expiry, before you begin the upgrade.
    You can find how much time the Oracle Key Vault server certificate has before it expires by checking the OKV Server Certificate Expiration setting on the Configure Alerts page in the Oracle Key Vault management console.
14. If you are performing an upgrade while using an HSM as a Root of Trust, then consult Oracle Key Vault Root of Trust HSM Configuration Guide for any additional steps that may be needed.
15. Ensure that the backup of the orapwdbfwdb file matches the original file.
    1. SSH into the Oracle Key Vault system as user support, then switch to user root:

       ssh support@<OKV_IP_Address>
       su – root
       

    2. Verify that the backup file exists:

       su - oracle
       ls -ltr /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

    3. If the backup file exists, then perform the following steps:
       • Compare the original file with the backup file:

         diff /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

       • If there is a difference between the files, then update the backup file by copying the original file:

         cp /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb

You can upgrade a pair of Oracle Key Vault servers in a primary-standby deployment.

• About Upgrading an Oracle Key Vault Server Primary-Standby Pair
  In a primary-standby deployment you must upgrade both primary and standby Oracle Key Vault servers.
• Upgrading a Pair of Primary-Standby Oracle Key Vault Servers
  You should allocate several hours to upgrade the primary server after upgrading the standby.

When you upgrade the Oracle Key Vault server software appliance, also upgrade the endpoint software to get access to the latest enhancements.

1. For the endpoint upgrade of a TDE-enabled database, the database instance must be shut down to install the latest PKCS#11 library. Oracle recommends upgrading all endpoints for TDE-enabled databases on the same host together. Review the instructions in step 6 before proceeding with the upgrade of endpoints for TDE-enabled databases.
   You can upgrade an endpoint by updating the endpoint software or by re-enrolling the endpoint. Perform steps 2 to 4 to update the endpoint software.

   Or

   Perform step 5 to re-enroll the endpoint.

2. Download the endpoint software (okvclient.jar) and install it in your existing endpoint directory path as follows:
   1. Go to the Oracle Key Vault management console login screen.
   2. Click the Endpoint Enrollment and Software Download link.
   3. In the Download Endpoint Software Only section, select the appropriate platform from the drop-down list.
   4. Click the Download button to download the okvclient.jar file.
3. Identify the path to your existing endpoint installation that you are about to upgrade. For example, /etc/ORACLE/KEYSTORES/okv (where /etc/ORACLE/KEYSTORES is WALLET_ROOT of your database, or the softlink in $ORACLE_BASE/okv/$ORACLE_SID points to).
4. Install the endpoint software by running the following command:

   java -jar okvclient.jar -d existing_endpoint_directory_path

   For example:

   java -jar okvclient.jar -d /etc/ORACLE/KEYSTORES/okv

   If you are installing the okvclient.jar file for an endpoint that has Oracle Database 23ai, then include the -arch db23ai option during the installation. The new endpoint software for Oracle Database 23ai is required to support new features such as using OpenSSL for FIPS mode and the new version of local auto login wallets in Oracle Database 23ai. The new endpoint software for Oracle Database 23ai is supported on the Linux-x64 platform only.

   For example:

   Java -jar okvclient.jar -d /home/oracle/okvutil -arch db23ai

5. Perform the following steps to re-enroll the endpoint software, which also generates a new endpoint certificate. The easiest way to re-enroll an endpoint is by using the following commands of the RESTful services utility:
   1. Re-enroll the endpoint by using the following RESTful services utility command:

      okv admin endpoint re-enroll

   2. Back up the OKV_HOME directory and delete the files under OKV_HOME:

      cp -R $OKV_HOME $OKV_HOME_bkp_date +%Y%m%d 

   3. Go to the $OKV_HOME directory and remove all the files.
   4. For Oracle Database 21c and earlier:
      Download and install the endpoint software by using the following RESTful services utility command:

      okv admin endpoint provision

      For Oracle Database 23ai:

      Download and install the endpoint software by using the following RESTful services utility command:

      okv admin endpoint provision --arch db23ai

   Re-enrolling an endpoint generates a new okvclient.jar file and installs the file in the OKV_HOME directory but maintains the relationship between the endpoint and its default wallet.

   Note:

   To re-enroll an endpoint without using RESTful services utility, follow the steps described in How to Re-enroll an Endpoint.
6. Install the updated PKCS#11 library file.
   This step is needed only for online TDE master encryption key management by Oracle Key Vault. If an endpoint uses online TDE master encryption key management by Oracle Key Vault, then you must upgrade the PKCS#11 library while upgrading the endpoint software.

   For Oracle Database 21c and earlier:

   Ensure that database instance is shut down before installing the PKCS#11 library in the location /opt/oracle/extapi/64/hsm/oracle/1.0.0.

   • On UNIX/Linux platforms: Run root.sh from the bin directory of endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints.

     $ sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh

     Or

     $ su - root
     # /etc/ORACLE/KEYSTORES/okv/bin/root.sh
     

   • On Windows platforms: Run root.bat from the bin directory of endpoint installation directory to copy the latest liborapkcs.dll file for Oracle Database endpoints. You will be prompted for the version of the database in use.

     bin\root.bat

   If you are upgrading multiple endpoints for TDE-enabled databases on a host, you must install the latest Oracle Key Vault PKCS#11 library only once on the host computer.

   On the host, perform the following steps when upgrading multiple endpoints for TDE-enabled databases:

   • Complete the upgrade of all Oracle Key Vault endpoints for the TDE-enabled databases.
   • Shut down corresponding TDE-enabled database instances.
   • Run the root.sh or root.bat script to install the latest Oracle Key Vault PKCS#11 library.

   For Oracle Database 23ai:

   Oracle recommends that you install the latest liborapkcs.so file in a fixed custom location using the root.sh script to enable upgrading the liborapkcs.so file in future without encountering database downtime.

   The fixed custom location is in the following format:

    /opt/oracle/extapi/64/pkcs11/okv/<okv_version>/lib

   For example, in the Oracle Key Vault 21.11 release, the location is

    /opt/oracle/extapi/64/pkcs11/okv/21.11.0.0.0/lib

   • On UNIX/Linux platforms: Run root.sh from the bin directory of the endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints.

     $ sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh –-okv_pkcs11_library_location

     Or

     $ su - root
     # /etc/ORACLE/KEYSTORES/okv/bin/root.sh –okv_pkcs11_library_location
     

   If you are upgrading multiple endpoints for TDE-enabled databases on a host, you must install the latest Oracle Key Vault PKCS#11 library only once for the endpoint software version.

   On the host where there are multiple endpoints for TDE-enabled Oracle database 23ai, each database can upgrade their endpoint software separately and switch to using the upgraded liborapkcs.so library by setting up the database initialization parameter PKCS11_LIBRARY_LOCATION to point to the upgraded library.

   Note:

   Installing the liborapkcs.so library in a legacy location is also supported with Oracle Database 23ai databases. However, Oracle does not recommend it.
7. Update the SDK software.
   If you have already deployed the SDK software, Oracle recommends that you redeploy the SDK software in the same location after you complete the upgrade to Oracle Key Vault release 21.10. This enables you to have access to the new SDK APIs that were introduced since the Oracle Key Vault version that you are upgrading from.
   1. Go to the Oracle Key Vault management console login screen.
   2. Click the Endpoint Enrollment and Software Download link.
   3. c. In the Download Software Development Kit section, select the appropriate language and platform for your site.
   4. Click the Download button to get the SDK zip file.
   5. Identify the existing location where SDK software was already deployed.
   6. Navigate to the directory in which you saved the SDK zip file.
   7. Unzip the SDK zip file.
      For example, on Linux, to unzip the Java SDK zip file, use the following command:

      unzip -o okv_jsdk.zip -d existing_endpoint_sdk_directory_path

      For the C SDK zip file, use this command:

      unzip -o okv_csdk.zip -d existing_endpoint_sdk_directory_path

   8. Do not exit this page.
8. If you had deployed the RESTful services utility in the previous release, then re-deploy the latest okvrestclipackage.zip file.

   The latest okvrestclipackage.zip file enables you to have access to the new RESTful services utility commands that were introduced since the Oracle Key Vault version that you are upgrading from.

   You can use wget or curl to download okvrestclipackage.zip.

   wget --no-check-
   certificate https://Oracle_Key_Vault_IP_address:5695/ okvrestclipackage.zip curl -O -
   k https://Oracle_Key_Vault_IP_address:5695/okvrestservices.jar

   Start the Oracle Databases if the upgrade of Oracle Key Vault endpoints for all of the TDE enabled databases on this host machine is complete.

   For Oracle Database 23ai which is using the custom path for installing the liborapkcs.so library from step 6 above, perform the following additional steps.

   1. Log in to the CDB Root as a user who has been granted the ALTER SYSTEM privilege.
   2. Set the static initialization parameter PKCS11_LIBRARY_LOCATION to point to the upgraded liborapkcs.so library.
      For example:

      ALTER SYSTEM SET
      PKCS11_LIBRARY_LOCATION=’/opt/oracle/extapi/64/pkcs11/okv/21.11.0.0.0/lib/liborapkcs.so’ SCOPE=SPFILE SID=’*’;

9. After the database restart, the database will switch to using the new upgraded library from the custom path. The database is also now set to switch to newer libraries in future without encountering downtime.

   After you upgrade the Oracle Key Vault to a new release in future (for example 21.xx.0.0.0), the root.sh file in the upgraded endpoint can be used to copy the liborapkcs.so library to the new path /opt/oracle/extapi/64/pkcs11/21.12.0.0.0/lib. The database can switch to the new library by using the SWITCHOVER command.

   For example:

   ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY 
   ‘/opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib/liborapkcs.so’ 
   FOR ALL CONTAINERS;

   At this stage, the endpoint is fully upgraded.

10. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Real Application Clusters (Oracle RAC) environment, then perform the following steps on each Oracle RAC node:
    1. Perform the endpoint upgrade on each Oracle RAC node.
    2. Set the extractable attribute value for symmetric keys.
       By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then you must set this value to false. You can set an extractable attribute value as follows:

       • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific setting overrides the global endpoint settings.
       • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
       • Modify the extractable attribute of an existing symmetric key.
       See Oracle Key Vault Administrator's Guide.
    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the Oracle RAC node. Use the following syntax:

       ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
       [FORCE KEYSTORE][USING TAG 'tag_name'] 
       IDENTIFIED BY [EXTERNAL STORE | keystore_password]
       [WITH BACKUP [USING backup_identifier']];

       See Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.

11. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Data Guard environment, then do the following on the primary and standby databases:
    1. Perform the endpoint upgrade on the primary and standby databases.
    2. Set the extractable attribute value for symmetric keys.
       By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then you must set this value to false. You can set an extractable attribute value as follows:

       • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific setting overrides the global endpoint settings.
       • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
       • Modify the extractable attribute of an existing symmetric key.
       See Oracle Key Vault Administrator's Guide.
    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the primary and standby databases. Use the following syntax:

       ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
       [FORCE KEYSTORE]
       [USING TAG 'tag_name'] 
       IDENTIFIED BY [EXTERNAL STORE | keystore_password]
       [WITH BACKUP [USING 'backup_identifier'']];

       See Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.

You must perform server backup and user password tasks after completing a successful upgrade.

1. Take a full backup of the upgraded Oracle Key Vault Server Database to a new remote destination. Avoid using the old backup destination for the new backups.
2. Schedule a new periodic incremental backup to the new destination defined in the preceding step.
3. Change the Oracle Key Vault administrative passwords.
   Password hashing has been upgraded to a more secure standard than in earlier releases. This change affects the operating system passwords, support and root. You must change Oracle Key Vault administrative passwords after the upgrade to take advantage of the more secure hash.