DCS-10001:Internal error encountered: Failed to install/update rpms

# odacli update-server

1. Rebuild the RPM database:

   mkdir /var/lib/rpm_backup
   cp -a /var/lib/rpm/__db* /var/lib/rpm_backup/
   rm -f /var/lib/rpm/__db*
   rpm -qa
   rpm --rebuilddb
   rpm -qa

2. Verify that the ipmitool is present on the system. If the following command returns no output, then it means that the ipmitool RPM is not installed.

   rpm -qa | grep ipmitool

3. If ipmitool is not present, then install the RPM as follows:

   rpm -ivh /opt/oracle/oak/pkgrepos/os/19.27/osrpms/hmpipmitool-1.8.18.0-29.el8.x86_64.rpm

4. Start Oracle Clusterware on all nodes. For high-availability systems, run the following command on both nodes:

   grid_home/crsctl start crs

5. Run the odacli update-server command again:

   odacli update-server -v 19.27.0.0.0

1. Rebuild the RPM database:

   mkdir /var/lib/rpm_backup
   cp -a /var/lib/rpm/__db* /var/lib/rpm_backup/
   rm -f /var/lib/rpm/__db*
   rpm -qa
   rpm --rebuilddb
   rpm -qa

2. Start Oracle Clusterware on all nodes. For high-availability systems, run the following command on both nodes:

   grid_home/crsctl start crs

3. Run the odacli update-server command again:

   odacli update-server -v 19.27.0.0.0

# odacli upgrade-dbsystem

ORA-01652: unable to extend temp segment by 128 in tablespace TEMP_ENC

Check: Tablespace Status - ERROR

# odacli update-dbhome 
# odacli update-database

alter database tempfile 4 resize 400M;
alter session set container=CHSTPDB;
alter database tempfile 5 resize 400M;

ProvDbHome by using RHP                  Failure

When describe the generated prepatch report, failed precheck item can be seen as below,

Evaluate DBHome patching with   Failed    DCS-10267 - failed to run the patch   
RHP                                       precheck with Oracle FPP for Oracle   
                                          home ID                               
                                          <UUID>  
                                          For input string: "<dbhome_name>" 

# odacli create-prepatchreport -d

DCS-10172:DCS infrastructure is not ready: The infrastructure is still initializing

# odacli update-server

1. On the DB system, stop the DCS agent service:

   # systemctl stop initdcsagent

2. Delete the entry HWADDR=null from the /etc/sysconfig/network-scripts/ifcfg-ib* configuration files:

   # sed -i '/HWADDR=null/d' /etc/sysconfig/network-scripts/ifcfg-ib*

3. Restart the network service:

   # systemctl restart network

4. Start the DCS agent service:

   # systemctl start initdcsagent

5. Wait for about 5 minutes and then verify that the DCS agent infrastructure is initialized, and both Oracle HAMI members are ONLINE.

   # /opt/oracle/dcs/hami/bin/hamictl.sh status

6. Verify that Oracle Clusterware service is online:

   # CRS_HOME/bin/crsctl check cluster -all
   **************************************************************
   Node0:
   CRS-4537: Cluster Ready Services is online
   CRS-4529: Cluster Synchronization Services is online
   CRS-4533: Event Manager is online
   **************************************************************
   Node1:
   CRS-4537: Cluster Ready Services is online
   CRS-4529: Cluster Synchronization Services is online
   CRS-4533: Event Manager is online
   **************************************************************

7. If Oracle Clusterware is not online on any DB system, then restart Oracle Clusterware on that DB system:

   # CRS_HOME/bin/crsctl stop crs -f
   # CRS_HOME/bin/crsctl start crs

# systemctl status kdump -l
 kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2024-10-15 11:51:15 IST; 8min ago
  Process: 6280 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
 Main PID: 6280 (code=exited, status=1/FAILURE)

Oct 15 11:51:12 systemd[1]: Starting Crash recovery kernel arming...
Oct 15 11:51:12 kdumpctl[6471]: kdump: No kdump initial ramdisk found.
Oct 15 11:51:12 kdumpctl[6471]: kdump: Rebuilding /boot/initramfs-5.4.17-2136.335.4.el8uek.x86_64kdump.img
Oct 15 11:51:13 kdumpctl[6566]: kdump: Warning: There might not be enough space to save a vmcore.
Oct 15 11:51:13 kdumpctl[6566]: kdump:          The size of /dev/mapper/VolGroupSys-LogVolRoot should be greater than 393610208 kilo bytes.
Oct 15 11:51:15 dracut[8055]: Executing: /usr/bin/dracut --add kdumpbase --quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode strict --hostonly-nics  -o "plymouth dash resume ifcfg earlykdump" --compress=xz --mount "/dev/mapper/VolGroupSys-LogVolRoot /sysroot ext4 rw,relatime,nofail,x-systemd.before=initrd-fs.target" --no-hostonly-default-device --add-device /dev/md0 -f /boot/initramfs-5.4.17-2136.335.4.el8uek.x86_64kdump.img 5.4.17-2136.335.4.el8uek.x86_64
Oct 15 11:51:15 kdumpctl[7997]: dracut: /lib/modules/5.4.17-2136.335.4.el8uek.x86_64//modules.dep is missing. Did you run depmod?
Oct 15 11:51:15 dracut[8055]: /lib/modules/5.4.17-2136.335.4.el8uek.x86_64//modules.dep is missing. Did you run depmod?
Oct 15 11:51:15 kdumpctl[6471]: kdump: mkdumprd: failed to make kdump initrd
Oct 15 11:51:15 kdumpctl[6471]: kdump: Starting kdump: [FAILED]
Oct 15 11:51:15 systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE
Oct 15 11:51:15 systemd[1]: kdump.service: Failed with result 'exit-code'.
Oct 15 11:51:15 systemd[1]: Failed to start Crash recovery kernel arming.

# odacli update-server

# systemctl restart kdump

# systemctl status kdump -l
kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2024-10-19 09:34:23 IST; 8s ago
  Process: 2028 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
 Main PID: 2028 (code=exited, status=0/SUCCESS)

Oct 19 09:34:21 dracut[2762]:  rd.lvm.lv=VolGroupSys/LogVolRoot
Oct 19 09:34:21 dracut[2762]:  rd.md.uuid=1e7140f4:2f5386a9:3093dd8d:ee3b9b29
Oct 19 09:34:22 dracut[2762]: *** Install squash loader ***
Oct 19 09:34:22 dracut[2762]: *** Squashing the files inside the initramfs ***
Oct 19 09:34:23 dracut[2762]: *** Squashing the files inside the initramfs done ***
Oct 19 09:34:23 dracut[2762]: *** Creating image file '/boot/initramfs-5.4.17-2136.335.4.el8uek.x86_64kdump.img' ***
Oct 19 09:34:23 dracut[2762]: *** Creating initramfs image file '/boot/initramfs-5.4.17-2136.335.4.el8uek.x86_64kdump.img' done ***
Oct 19 09:34:23 kdumpctl[2104]: kdump: kexec: loaded kdump kernel
Oct 19 09:34:23 kdumpctl[2104]: kdump: Starting kdump: [OK]
Oct 19 09:34:23 systemd[1]: Started Crash recovery kernel arming.

-----------------------
2024-07-29 14:24:30,351 WARN [backgroundjob-zookeeper-pool-7-thread-2] [] o.j.s.JobZooKeeper: JobRunr encountered a problematic exception. Please create a bug report (if possible, provide the code to reproduce this and the stacktrace) - Processing will continue.
java.lang.NullPointerException: null
        at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.pollIntervalInSecondsTimeBoxIsAboutToPass(ZooKeeperTask.java:93)
        at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.getJobsToProcess(ZooKeeperTask.java:84)
        at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.processJobList(ZooKeeperTask.java:57)
        at org.jobrunr.server.zookeeper.tasks.ProcessOrphanedJobsTask.runTask(ProcessOrphanedJobsTask.java:29)
        at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.run(ZooKeeperTask.java:47)
        at org.jobrunr.server.JobZooKeeper.lambda$runMasterTasksIfCurrentServerIsMaster$0(JobZooKeeper.java:76)
        at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
        at org.jobrunr.server.JobZooKeeper.runMasterTasksIfCurrentServerIsMaster(JobZooKeeper.java:76)
        at org.jobrunr.server.JobZooKeeper.run(JobZooKeeper.java:56)
-----------------------

# odacli update-server

1. Restart the DCS agent:

   systemctl restart initdcsagent

2. Verify that the DCS agent is running:

   odacli ping-agent
   odacli list-jobs
   odacli describe-component

# odacli upgrade-database -i 16288932-61c6-4a9b-beb0-4eb19d95b2bd -to b969dd9b-f9cb-4e49-8e0d-575a0940d288
DCS-10001:Internal error encountered: dbStorage metadata not in place:
DCS-12013:Metadata validation error encountered: dbStorage metadata missing
Location info for database database_unique_name..

# odacli upgrade-database

1. Verify that the odacli list-dbstorages command displays null for the redo location for the database that reported the error. For example, the following output displays a null or empty value for the database unique name F.

   # odacli list-dbstorages
   
   ID                                     Type   DBUnique Name  Status     
   Destination Location  Total      Used       Available      
   ---------------------------------------- ------ --------------------
   ...
   ...
   ...
   198678d9-c7c7-4e74-9bd6-004485b07c14     ASM    F            CONFIGURED   
   DATA    +DATA/F  4.89 TB    1.67 GB    4.89 TB                                                                   
   REDO    +REDO/F  183.09 GB  3.05 GB    180.04 GB                                                                                
   RECO             8.51 TB              
   ...
   ...
   ...

   In the above output, the RECO record has a null value.

2. Manually create the RECO directory for this database. If the database unique name is dbuniq, then run the asmcmd command as the grid user.

   asmcmd

3. Run the mkdir command.

   asmcmd> mkdir +RECO/dbuniq

4. Verify that the odacli list-dbstorages command output does not display a null or empty value for the database.
5. Rerun the odacli upgrade-database command.

Failed to execute sqlpatch for database …

# odacli update-database

1. Run the following SQL*Plus command:

   alter system set nls_sort='BINARY' SCOPE=SPFILE;

2. Restart the database using srvctl command.
3. Retry applying the datapatch with dbhome/OPatch/datapatch -verbose -db dbUniqueName.

Verify the Alternate Archive    Failed    AHF-4940: One or more log archive 
Destination is Configured to              destination and alternate log archive
Prevent Database Hangs                    destination settings are not as recommended           

1. Run the odacli update-dbhome command with the -f option.

   /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid 7c67c5b4-f585-4ba9-865f-c719c63c0a6e -v 19.27.0.0.0 -f

One or more log archive destination and alternate log archive destination settings are not as recommended 
Software home check failed 

odacli update-dbhome -j -v 19.27.0.0.0 -i dbhome_id -sko

# time odacli update-dcscomponents -v 19.27.0.0.0 
^[[ADCS-10008:Failed to update DCScomponents: 19.27.0.0.0
Internal error while patching the DCS components : 
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer  
to /opt/oracle/dcs/log/-dcscomponentsPreCheckReport.log on node 1 for  
details.   

DCS-10001:Internal error encountered: Server upgrade state is not NORMAL node_name 

# odacli modify-database -n dbname -ha

DCS-10001:Internal error encountered: Job 'Provision DB System 'db_system_name'' failed. 
DCS-10001:Internal error encountered: Chronyd failed to sync the clock. :

odacli create-appliance -r path_to_json_payload
odacli create-dbsystem -p path_to_json_payload

DCS-10001:Internal error encountered: Job 'Provision DB System 'db_system_name'' failed. 
DCS-10001:Internal error encountered: Chronyd failed to sync the clock. :

odacli create-appliance -r path_to_json_payload
odacli create-dbsystem -p path_to_json_payload

1. Run the provisioning job using the IP address of one NTP server.

   # /sbin/chronyd -Q "server 1.1.1.1 iburst"

2. If the server is reachable and valid, the chronyd example output will show the difference between the system clock and the NTP clock:

   2025-04-03T20:50:11Z System clock wrong by -0.000065 seconds (ignored)

3. Check the validity of the agent certificates.

   # openssl x509 -noout -in /opt/oracle/dcs/odamysqlcert/client/dcsagent-client-cert.pem -startdate -enddate
   notBefore=May 15 02:11:13 2025 GMT
   notAfter=Mar 24 02:11:13 2035 GMT

4. If the correct time falls out of the valid dates from the certificates, follow these steps if you are provisioning a bare metal system:
   1. Stop the agent and MySQL services.

      # systemctl stop initdcsagent.service
      # systemctl stop oda-mysql.service

   2. Manually set the clock on both nodes:

      # date --set="correct_date"

   3. Make a backup of the existing MySQL certificates on both nodes.

      # mv /opt/oracle/dcs/odamysqlcert /opt/oracle/dcs/odamysqlcert_old

   4. Run the command /opt/oracle/dcs/mysql/cert/gencerts.sh.
   5. Confirm that the new certificates have the correct dates on both nodes.

      # openssl x509 -noout -in /opt/oracle/dcs/odamysqlcert/client/dcsagent-client-cert.pem -startdate -enddate
      notBefore=May 15 02:11:13 2025 GMT
      notAfter=Mar 24 02:11:13 2035 GMT

   6. Start the MySQL and agent services on both nodes.

      # systemctl stop initdcsagent.service
      # systemctl stop oda-mysql.service

   7. Confirm that the agent is working.

      # odacli describe-component

   8. Proceed with the provisioning.
5. Follow these steps if you are provisioning a DB system:
   1. Make sure the bare metal system has the correct date on both nodes. If not, verify the bare metal system certificates will not become invalid after adjusting the clock. If the certificates areinvalid, follow step 4 to generate new bare metal system certificates. If the certificates are valid, fix the date on both bare metal system nodes, then restart the services:

      # date --set="correct date"
      # systemctl restart oda-mysql.service
      # systemctl restart initdcsagent.service

   2. Proceed with the DB system creation.

DCS-10001:THE CONNECTION IS CLOSED

odacli create-dbsystem

ORA-16072: a minimum of one standby database destination is required

terminating the instance due to ORA error 16072

# odacli modify-asmport

[jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - [FATAL] Error in Process: /u01/app/KvEl6/product/19.0.0.0/dbhome_2/bin/orapwd
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - Enter password for SYS:
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - OPW-00010: Could not create the password file.
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - ORA-00600: internal error code, arguments: [kfzpCreate02], [0], [], [], [], [], [], [], [], [], [], []
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - ORA-15260: permission denied on ASM disk group
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 679
    [jobid-74f31148-ebe0-4507-9296-b9ad4ca7e03b] - ORA-06512: at line 2

PRCZ-4001 : failed to execute command "/u01/app/6RXNI/product/19.0.0.0/dbhome_15//bin/dbca" using the privileged execution plugin "odaexec" on nodes "scaoda901c7n1" within 5,000 seconds
PRCZ-2103 : Failed to execute command "/u01/app/6RXNI/product/19.0.0.0/dbhome_15//bin/dbca" on node "scaoda901c7n1" as user "6RXNI". Detailed error:
[FATAL] [DBT-05801] There are no ASM disk groups detected.
  CAUSE: ASM may not be configured, or ASM disk groups are not created yet.
  ACTION: Create ASM disk groups, or change the storage location to File System.
[FATAL] [DBT-05801] There are no ASM disk groups detected.
  CAUSE: ASM may not be configured, or ASM disk groups are not created yet.
  ACTION: Create ASM disk groups, or change the storage location to File System.

# odacli create-database

DCS-10045:Validation error encountered: Registration failed : Please check the agent logs for details.

# odacli configure-asr

DCS-10001:Internal error encountered: Failed to clear all listeners from database

# odacli create-database

CRS-2672: Attempting to start 'vm_name.kvm' on 'oda_server'
CRS-5017: The resource action "vm_name.kvm start" encountered the following
error:
CRS-29200: The libvirt virtualization library encountered the following
error:
Timed out during operation: cannot acquire state change lock (held by
monitor=remoteDispatchDomainCreate)
. For details refer to "(:CLSN00107:)" in
"/u01/app/grid/diag/crs/<oda_server>/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'vm_name.kvm' on 'oda_server' failed
CRS-2679: Attempting to clean 'vm_name.kvm' on 'oda_server'
CRS-2681: Clean of 'vm_name.kvm' on 'oda_server' succeeded
CRS-4000: Command Start failed, or completed with errors.

ORA-15333: disk is not visible on client instance

systemctl restart initdcsagent 

SQL> alter system checkpoint;

DCS-10033:Service DCS agent is down.

# odacli ping-agent

1. Validate that the issue is due to interconnect not working. From the first node, run the command:

   # arping -I icbond0 192.168.16.25 -c 10

   The output is similar to the following:

   ARPING 192.168.16.25 from 192.168.16.24 icbond0
   Sent 10 probes (10 broadcast(s))
   Received 0 response(s)

2. On both nodes, modify the /etc/sysconfig/network-scripts/ifcfg-icbond0 file to add arp_interval=100 to BONDING_OPTS. The update is as follows:

   BONDING_OPTS="mode=active-backup miimon=100 primary=p1p1 arp_interval=100"

3. On both nodes, restart the network:

   # systemctl restart network

4. On both nodes, restart the agent and wait for a few minutes:

   # systemctl restart initdcsagent

DCS-10001:Internal error encountered: Unable to update dg config

"Error: ORA-16532: Oracle Data Guard broker configuration does not exist."

One or more pre-checks failed for [DB]

# odacli create-preupgradereport 

# odacli describe-preupgradereport 

"The following services [TDG1yn_ro, TDG1yn_rw, Y6Z_ro, Y6Z_rw] created on database 
'TDG1yn' can result in a failure in 'detach-node'

1. Stop the service reported:

   srvctl stop service -d db_unique_name -service service_name

2. Remove the service:

   srvctl remove service -d db_unique_name -service service_name

"DCS-10001:Internal error encountered: Job 'Update Data Guard status (Primary site)' failed with error: 
DCS-10001:Internal error encountered: Unable to update dg config"

odacli configure-dataguard

1. Check that Oracle Data Guard was successfully configured for two standby databases:

   check DGMGRL>SHOW CONFIGURATION; 

2. Check Oracle Data Guard status on primary and all standby sites:

   odacli list-dataguardstatus -j

3. Delete Oracle Data Guard status if it does not have information about all standby sites:

   DEVMODE=true odacli delete-dataguardstatus -i dataguardstatus_id

4. Check that Oracle Data Guard status is correct:

   odacli list-dataguardstatus -j

5. Run the odacli create-dataguardstatus command on both standby sites from the active node of the database.

   DEVMODE=true odacli create-dataguardstatus -i database_id -r config_dg.json -n dataguardstatus_id_of_primary

   Example of config_dg.json file:

   {
     "name": "dgname",
     "protectionMode": "MAX_PERFORMANCE",
     "enableFlashback": true,
     "enableActiveDg": false,
     "replicationGroups": [
       {
         "sourceEndPoints": [
           {
             "endpointType": "PRIMARY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "primary",
             "serviceName": "primary.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           },
       {
         "endpointType": "PRIMARY",
         "hostName": "xxx.com",
         "listenerPort": 1521,
         "databaseUniqueName": "primary",                 
         "serviceName": "primary.com",
         "sysPassword": "***",
         "ipAddress": "x.x.x.x"
       }
         ],
         "targetEndPoints": [
           {
             "endpointType": "STANDBY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "standby1",        
             "serviceName": "standby1.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           },
           {
             "endpointType": "STANDBY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "standby1",
             "serviceName": "standby1.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           }
         ],
         "transportType": "ASYNC"
       },
           {
         "sourceEndPoints": [
           {
             "endpointType": "PRIMARY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "primary",
             "serviceName": "primary.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           },
       {
         "endpointType": "PRIMARY",
         "hostName": "xxx.com",
         "listenerPort": 1521,
         "databaseUniqueName": "primary",                 
         "serviceName": "primary.com",
         "sysPassword": "***",
         "ipAddress": "x.x.x.x"
       }
         ],
         "targetEndPoints": [
           {
             "endpointType": "STANDBY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "standby2",        
             "serviceName": "standby2.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           },
           {
             "endpointType": "STANDBY",
             "hostName": "xxx.com",
             "listenerPort": 1521,
             "databaseUniqueName": "standby2",
             "serviceName": "standby2.com",
             "sysPassword": "***",
             "ipAddress": "x.x.x.x"
           }
         ],
         "transportType": "ASYNC"
       }
     ]
   }

DCS-10001:Internal error encountered: Error creating job 'Delete Dataguard Status(Standby site)': com.oracle.pic.commons.client.exceptions.RestClientException: DCS-10000:Resource dgConfig with ID xxx is not found

# odacli modify-database -n dbname -rkt -tn target_node_name

# odacli delete-database -n db_name

1. Log into as the Oracle Key Vault administrator to the Oracle Key Vault server where the Oracle Key Vault wallet is present.
2. Navigate to the Keys & Wallets tab.
3. Click the edit icon for the wallet that you want to delete.
4. In the Select Endpoint/User Group section, select the Type as Users from the drop down list.
5. Select the user that owns the Oracle Key Vault wallet.
6. In the Select Access Level section, select Read and Modify, and then Manage Wallet.
7. Click Save.
8. Delete the database.

Failed to execute sqlpatch for database …

# odacli delete-dbhome

SQL> ALTER USER SYS ACCOUNT LOCK;
ALTER USER SYS ACCOUNT LOCK
*
ERROR at line 1:
ORA-40365: The SYS user cannot be locked while the password file is in its
current format.

1. Get the current password file location of the database.

   srvctl config database -d dbUniqueName | grep -i password
   
   +DATA/dbUniqueName/PASSWORD/pwddbnamenumbers

2. Set the password file location of the database to empty.

   srvctl modify database -d dbUniqueName -pwfile ''

3. Recreate the password file location of the database from the existing password file in step1.

   orapwd file='new_password_file_location' format=12.2 input_file='output_in_step_1' dbuniquename=dbUniqueName

   output_in_step_1 is similar to +DATA/dbUniqueName/PASSWORD/pwddbnamenumbersnew_password_file_location can be +DATA/dbUniqueName/PASSWORD/orapwdbname , or another Oracle ASM location preferably under +DATA/dbUniqueName

4. Confirm the new password file is in 12.2 format.

   orapwd describe file='step3_new_password_file_location'
   Password file Description : format=12.2 

5. Set the password file location of the database to the new location in step 3 with srvctl.

   srvctl modify database -d dbUniqueName -pwfile 'step3_new_password_file_location'

6. Check to see if previous password can still be used to login to the database.

   sqlplus sys/"password"@dbUniqueName as sysdba

7. Lock SYS user.

   SQL> alter user sys account lock; 
   User altered.
   
   [root@n1 ~]# odacli list-databases
   333cd996-4de4-472c-a290-7907b0bd8313     ccc        RAC      19.26.0.0.250121     false   OLTP     EE       odb1     ASM      CONFIGURED   572a21bb-5912-4bad-a217-34611c821a89 
   ...
   
   [root@n1 ~]# odacli describe-database -n ccc -j
   {
     "id" : "333cd996-4de4-472c-a290-7907b0bd8313",
     "name" : "ccc",
     "dbName" : "ccc",
     "databaseUniqueName" : "ccc1",
   ...
   
   [root@n1 ~]# odacli describe-dbhome -i 572a21bb-5912-4bad-a217-34611c821a89 
   DB Home details                                                   
   ----------------------------------------------------------------
                        ID: 572a21bb-5912-4bad-a217-34611c821a89
                      Name: OraDB19000_home1
                   Version: 19.26.0.0.250121
             Home Location: /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1
   
   
   [root@n1 ~]# su - oracle
   [oracle@n1 ~]$ . oraenv
   ORACLE_SID = [oracle] ? ccc1
   ORACLE_HOME = [/home/oracle] ? /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1
   The Oracle base has been set to /u01/app/odaorabase/oracle
   
   
   [oracle@n1 ~]$ srvctl config database -d ccc1 | grep -i password
   Password file: +DATA/CCC1/PASSWORD/pwdccc1.305.1201883837
   [oracle@n1 ~]$ 
   [oracle@n1 ~]$ srvctl modify database -d ccc1 -pwfile ''
   [oracle@n1 ~]$ orapwd file='+DATA/CCC1/PASSWORD/orapwccc' format=12.2 input_file='+DATA/CCC1/PASSWORD/pwdccc1.305.1201883837' dbuniquename=ccc1
   [oracle@n1 ~]$ orapwd describe file='+DATA/CCC1/PASSWORD/orapwccc'
   Password file Description : format=12.2 
   [oracle@n1 ~]$ srvctl modify database -d ccc1 -pwfile '+DATA/CCC1/PASSWORD/orapwccc'
   [oracle@n1 ~]$ sqlplus sys/"password"@ccc1 as sysdba
   SQL> alter user sys account lock; 
   
   User altered.

odacli configure-dataguard

1. Stop the NFS service on both nodes:

   service nfs stop

2. Clean up the bare metal system. See the Oracle Database Appliance Deployment and User's Guide for your hardware model for the steps.

1. Restart standby database in upgrade mode:

   srvctl stop database -d <db_unique_name> 
   Run PL/SQL command: STARTUP UPGRADE; 

2. Continue the enable apply process and wait for log apply process to refresh.
3. After some time, check the Data Guard status with the DGMGRL command:

   SHOW CONFIGURATION;