Guidelines for Updating SSL Keys and Certificates

If the certificate that the server uses is going to expire, or is no longer valid, you may need to replace the SSL key and certificate. This section describes the procedure to complete this task.

These directions describe creating a self-signed certificate, and an associated key, which is the default for Oracle NoSQL Database. Alternatively, you can use an external certificate, as described in Guidelines for Configuring External Certificates for an Existing Default Secure Installation.

You can update SSL Keys and certificates using one of the following two methods:

Use the plan update-tls-credentials command.
Use the manual process in each of the SNs to create a new SSL key and certificate, merge the truststore entries, copy the store.keys file to the security directory, and stop and start each SN sequentially as part of a rolling restart, if needed. Restarting the SNs will now only be needed if the data store has not yet been fully upgraded to Oracle NoSQL Database 24.4 or later.

Use the plan command

The plan update-tls-credentials command retrieves and installs the credential updates to the set of shared TLS (Transport Layer Security, earlier known as SSL) credentials used by Storage Node Agents (SNA) in the data store. You should use this plan only with data stores where all SNAs share the same credentials, and not for data stores with host-specific credentials.

plan update-tls-credentials [-retrieve-only|-install-only] [-force]

The plan command can be used in multiple ways depending on your requirements.

You can manually copy the new credentials and use the plan update-tls-credentials command with the -install-only flag. This is the best choice which you can use by default.
If you need to automate the entire process of updating SSL credentials and keys, you could use the plan update-tls-credentials command without specifying any options or flags. Then the plan command retrieves the credentials and installs them.
You could use the plan update-tls-credentials command with the -retrieve-only flag if you just want to retrieve the credentials and plan to install it later.

See plan update-tls-credentials for more details on how the SSL Keys and certificates are updated automatically using this plan command.

Manual process of updating SSL Keys and certificates

Updating the SSL key/certificate involves several steps:

Create a new key/certificate pair on a storage node.
Copy the new key/certificate pair to every storage node and merge the new certificate into the existing trust store files: client.trust and store.trust.
Restart each storage node sequentially, if needed. You will need to restart the SNs only if the data store has not yet been fully upgraded to Oracle NoSQL Database 24.4 or later. Once all SNs have been upgraded, they will notice the key and certificate changes without needing a restart.
Copy the client.trust with the merged entries to each of the clients.
Copy the store.keys that has the merged entries to each of the storage nodes, and restart each storage node sequentially, a second time. You need this step only for SNs running Oracle NoSQL Database 24.3 or earlier.
Remove the old certificate in store.trust in all the storage nodes.
Verify that only the new certificate is in use.

Complete these steps to update the SSL keys and certificates on a running store. Oracle NoSQL Database can remain operational throughout the entire process.

Note:

The Oracle NoSQL Database environment used below is deployed on 3 Storage Nodes with capacity=3 and Replication Factor (RF)=3.

For more information on security configuration files, see Security Configuration.

Create a New SSL Key Certificate

In the first Storage Node (SN1), create a temporary directory to store the key.
```
cd /users/user_name/tmp/kvroot/  
mkdir newKey 
```
On SN1 run the securityconfig utility to create a new key/certificate pair in the new directory, newKey. The new configuration needs to specify the same keystore password as your current configuration. If you do not specify a password with the -kspwd option, the utility prompts you to set a password.
```
java -jar $KVHOME/lib/kvstore.jar securityconfig \
config create -root /users/user_name/tmp/kvroot/newKey  -kspwd 123456
```
```
cd /users/user_name/tmp/kvroot/newKey
ls -R security
```
Output:
```
./security:client.security
security.xml store.trust temp.cert client.trust store.keys store.wallet
./security/store.wallet:
cwallet.sso
```
On SN1, merge the truststore entries using the config merge-trust command, as follows. After running this command, the client.trust and store.trust files will have two SSL certificate entries.
```
java -jar $KVHOME/lib/kvstore.jar securityconfig \
config merge-trust  -root $KVROOT -source-root \
/users/user_name/tmp/kvroot/newKey
```
Use keytool -list to list the entries in the keystore.
```
cd $KVROOT/security
keytool -list -keystore store.trust  
Enter keystore password: ********* 
```
Output:
```
Keystore type:
JKS Keystore provider: SUN
Your keystore contains 2 entries  
mykey_2, Feb 6, 2024,trustedCertEntry,  
Certificate fingerprint (SHA1):  
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A  
mykey, Feb 6, 2024,trustedCertEntry,  
Certificate fingerprint SHA1):  
89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67
```
Note:
The client.trust file also contains the same two entries as store.trust file shown above.

In a multiple Storage Node deployment, you must copy the new configuration (the security directory and its contents) to each Storage Node host's new configuration directory and run merge-trust as described on each host.
In SN2 and SN3, do the following:
```
cd /users/user_name/tmp/kvroot/  
mkdir newKey
```
Copy the contents of the directory newKey from SN1 to the /users/user_name/tmp/kvroot/newKey directory in SN2 and SN3.
In SN2 and SN3, merge the truststore entries using the config merge-trust command as shown below.
```
java -jar $KVHOME/lib/kvstore.jar securityconfig  \
config merge-trust -root $KVROOT \ 
-source-root /users/user_name/tmp/kvroot/newKey 
```
You can optionally list the entries in the keystore in SN2 and SN3 using keytool command (as shown above for SN1).
In an Oracle NoSQL Database running with a self-signed certificate, the client-side application will be able to connect to the data store with either the old or the new credentials once they switch to using the merged client truststore. It is a good practice to modify the clients at this point in the update procedure because that will allow the clients to connect to the data store now and throughout the update process. You need to copy the client.trust file with the merged entries (2 certificate entries) to each of the clients replacing the existing client.trust file used by the client applications. Once the update is complete, clients should switch to using the new truststore, so that the old certificates are cleared.
```
scp <SN1_host>:$KVROOT/security/client.trust \
<client_host>:<directory_client.trust>/client.trust
```
Note:
The client.trust is used to authenticate client-server communication, and store.trust to authenticate server-server communication.
Stop and start each Storage Node sequentially, making sure that each SN is completely up before restarting the next SN. Restarting one SN at a time is necessary because the data store must maintain the quorum during the restart of Storage Nodes while the SSL certificate is updated.
After the restart, the Storage Nodes will load both the new and old certificates from the merged store.trust, so that the Storage Nodes can recognize both the new and old SSL credentials. The Storage Nodes that are not yet restarted will continue to use the old SSL credentials from store.trust file.
In each of the Storage Nodes (SN1, SN2 and SN3), perform the following steps:
```
java -jar $KVHOME/lib/kvstore.jar stop -root $KVROOT  
java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT&
```
Start the Admin CLI, and check that all Replication Nodes (RNs) are up using the ping command:
```
java -jar $KVHOME/lib/kvstore.jar runadmin -host $HOSTNAME -port 5000 \
-security $KVROOT/security/client.security 
```
Logged in admin as anonymous
```
kv-> ping
```
Note:
You need to restart the SNs only if the data store has not yet been fully upgraded to Oracle NoSQL Database 24.4 or later. Once all SNs have been upgraded, they will notice the key and certificate changes without needing a restart.
The store.keys file contains the generated private key. The merge-trust utility used above merges only the certificates in the store.trust, but does not merge the private keys. To make the NoSQL Database use the new SSL private key, the new store.keys needs to be copied to the security directory under $KVROOT in every Storage Node as shown below.
In each of the Storage Nodes (SN1, SN2 and SN3), perform the following steps:
Copy the store.keys file to the security directory, stop the Storage Node and start it again. Stop and start each storage node sequentially as a rolling restart, making sure that each SN is completely up before restarting the next SN. Restarting one SN at a time is necessary because the data store must maintain the quorum during the restart.
```
cp /users/user_name/tmp/kvroot/newKey/security/store.keys $KVROOT/security/. 
java -jar $KVHOME/lib/kvstore.jar stop -root $KVROOT  
java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT&
```
Note:
You need to restart the SNs only if the data store has not yet been fully upgraded to Oracle NoSQL Database 24.4 or later. Once all SNs have been upgraded, they will notice the key and certificate changes without needing a restart.
On each Storage Node, remove the obsolete certificate mykey in store.trust. Then, rename the new certificate mykey_2 to mykey.
In each of the Storage Nodes ( SN1, SN2 and SN3), perform the following steps:
Remove the old certificate named mykey.
```
keytool -delete -keystore $KVROOT/security/store.trust -alias mykey  
Enter keystore password:  
```
Rename the newly created certificate, mykey_2, to the previous key's name, mykey.
```
keytool -changealias -keystore $KVROOT/security/store.trust \
-alias mykey_2 -destalias mykey
```
Only one key now exists, which is the newly generated one, called mykey. Verify that the new certificate is the only one listed using the following command:
```
keytool -list -keystore $KVROOT/security/store.trust  
Enter keystore password:
```
```
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry  
mykey, Feb 6, 2024,trustedCertEntry,  
Certificate fingerprint (SHA1):  
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A
```
The SSL keys and certificates have now been updated on the data store.

Additional verification while updating SSL Keys and Certificates

When you update SSL keys and certificates for a data store, these additional checks are performed. These checks are intended to detect potential problems with the new credentials.

The passwords for the keystore and truststore files must be the same as the ones for the keystore and truststore currently in use by the data store.
The types of the keystore and truststore files that you specified using the keystoreType and truststoreType parameters must match the exact types of the files.
The keystore must have an entry for the alias that you specified using the keystoreSigPrivateKeyAlias parameter. The alias identifies the keypair used to create signatures. The certificate corresponding to this key pair must be validated successfully with the certificate in the truststore. In this case, the keystore and truststore need to contain the same certificate although, the truststore can contain more than one certificate (For example, if it is part of a certificate chain).
The truststore must have an entry for the alias that you specified using the truststoreSigPublicKeyAlias parameter. This alias identifies the certificate used to verify signatures. The associated certificate must be validated successfully and match the certificate for the keystoreSigPrivateKeyAlias entry in the keystore.
The keystore must have entries for all of the serverKeyAlias parameters that you specified for any transport type. This alias identifies the keypair used by the store services. If you have not specified anything, the alias shared is used. The certificate corresponding to this key pair must be validated successfully with the certificate in the truststore.

Note:
The truststore can contain multiple certificates (For example, if it is part of a certificate chain). Additionally, the validation of the keystore certificate may also require multiple certificates in the truststore.
All serverKeyAlias entries must satisfy the verification requirements of the associated clientIdentityAllowed parameter, if any.
For transports for which the parameter clientAuthRequired is true, the keystore must have entries for all of the clientKeyAlias parameters that you specified. The clientkeyAlias parameter identifies the keypair used by either a direct connect Java client or a proxy. If you have not specified anything, the alias shared is used. The certificate corresponding to this key pair must be validated successfully with the certificate in the truststore.
All clientKeyAliasentries must satisfy the verification requirements of the associated serverIdentityAllowed parameter, if any.