Oracle Database Appliance X7-2S and X7-2M have NVMe storage configurations with storage expansion options.

Depending on the available drives, you can expand Oracle Database Appliance X7-2M storage to add NVMe disks or replace existing NVMe disks.

Use the ODAADMCLI commands to perform appliance storage maintenance tasks, including perform storage diagnostics and collect diagnostic logs for storage components.

Understand how you can replace existing NVMe disks on Oracle Database Appliance.

Oracle Database Appliance X7-2-HA platforms have options for high performance and high capacity storage configurations.

Oracle Database Appliance X7-2-HA are shipped with the base configuration of 16 TB SSD raw storage for DATA and 3.2 TB SSD raw storage for REDO, leaving 15 available slots to expand the storage. If you choose to expand the storage, you can fill the 15 slots with either SSD or HDD drives. For a high performance configuration, you can expand storage by adding 15 SSDs. If you want to add 15 HDDs, then the high performance configuration changes to a high capacity configuration. In this case, you must reimage and redeploy the appliance.

In all configurations, the base storage and the storage expansion shelf each have four (4) 800 GB SSDs for REDO disk group and five (5) 3.2 TB SSDs (either for DATA/RECO in the SSD option or FLASH in the HDD option).

High Performance

A high performance configuration uses solid state drives (SSDs) for DATA and REDO storage. The base configuration has 16 TB SSD raw storage for DATA and 3.2 TB SSD raw storage for REDO.

You can add up to fifteen 7.68 TB SSDs (available in five-packs). Note that 3.2 TB SSDs are no longer available. To support 7.68 TB SSDs in the system, ensure that your deployment is on Oracle Database Appliance release 18.7 or later.

Note:

With Oracle Database Appliance release 18.8, you can add 7.68 TB SSDs to a configuration with existing 3.2 TB SSDs. The 7.68 TB SSDs are partitioned down to match the 3.2 TB SSD capacity. The 3.2 TB SSDs and the expansion shelf are no longer available. However, if you replace all your existing 3.2 TB SSDs with 7.68 TB SSDs, then the entire 7.68 TB capacity of the SSDs is utilized for storage.

Description of the illustration ssd.png

High Capacity

A high capacity configuration uses a combination of SSD and HDD drives.

The base configuration has 16 TB SSD raw storage for FLASH disk group and 3.2 TB SSD raw storage for REDO.

With Oracle Database Appliance release 18.8, the following expansion options are available:

• Base shelf: Additional fifteen 14 TB HDDs (available in a fifteen-pack). Note that 10 TB HDDs are no longer available. To support 14 TB HDDs in the system, ensure that your deployment is on Oracle Database Appliance release 18.7 or later.

• Storage Expansion shelf: The expansion shelf is no longer available.

Note:

When you expand storage to include HDD on the base storage shelf, you must reposition the drives to the correct slots and redeploy the appliance after adding the HDD drives.

Note:

10TB HDDs are no longer available. To expand storage, you can use 15 packs of 14TB HDD drive.

Description of the illustration hdd.png

Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.

1. Update Oracle Database Appliance to the latest Patch Bundle before expanding storage.
2. Confirm both nodes are at the same version and patch bundle level for software and firmware.

   # odacli describe-component  

3. Check the disk health of the existing storage disks.

   Run the check on both nodes and use the default checks option to check the NetworkComponents, OSDiskStorage, SharedStorage, and SystemComponents.

   # odaadmcli validate -d

4. Run the odaadmcli show diskgroup command on each node to display and review Oracle Automatic Storage Management (Oracle ASM) disk group information.

   # odaadmcli show diskgroup DATA

   # odaadmcli show diskgroup RECO

    # odaadmcli show diskgroup REDO 

5. Confirm Oracle ASM and CRS health on both nodes.
   Run orachk on each node. If there is a problem connecting to either node, then check the /etc/bashrc  file and remove (or remark out) any values in the profile for root ; oracle ; grid users 
6. Confirm communications between the nodes and that SSH is working using the same password for oracle, root and grid.
   From each node:
   1. ssh to both nodes.
   2. Ping both nodes.
7. Confirm there is at least 10 GB of space available on each node.

   [root@oda]# df -h

   [root@odb]# df -h 

Add a pack of solid-state drives (SSDs) for data storage into the existing Oracle Database Appliance base configuration to fully populate the base storage shelf.

1. Insert disks one at a time in the slots.

   To add one 5-pack of SSDs, insert the disks in slots 5 to 9. To add two 5-pack of SSDs, insert the disks in slots 5 to 14. To add three 5-pack of SSDs, insert the disks in slots 5 to 19.

   Note:

   Allow at least one minute between inserting each disk.
   After all disks are added, go to Step 2.
2. Run the odaadmcli show ismaster command to determine which node is the master.

   # odaadmcli show ismaster

3. Run the odaadmcli expand storage command on the master node.

   #odaadmcli expand storage -ndisk number of disks to be added 
   -enclosure enclosure number of the disks to be added, either 0 or 1
   

   For example:

   
   #odaadmcli expand storage -ndisk 5 -enclosure 0
   Running precheck, it may take a few minutes.
   Precheck passed.
   Check the progress of expansion of storage by running 'odaadmcli show disk'
   Waiting for expansion to finish. It may take several minutes to complete
   depending upon the number of disks being expanded.
    
   odaadmcli expand storage command completed.
   Check status of newly expanded disk in OAK by running 'odaadmcli show disk'
   All of them must be ONLINE status. Check status of newly expanded disk in ASM
   by running 'asmcmd lsdsk -p', mount status must be CACHED and header status
   should be MEMBER for all the newly added disks. If the status of newly added
   disks in OAK and ASM are different then expand storage command failed.

   It takes 10 to 12 minutes to add all of the disks to the configuration.
4. Run the odaadmcli show disk command to ensure that all disks are listed, are online, and are in a good state.

   # odaadmcli show disk

5. Verify that the disks in slots 5 to 9 are added to Oracle Automatic Storage Management (Oracle ASM).
   1. Run the asm_script to verify that the disks in slots 5 to 9 are added to Oracle Automatic Storage Management (Oracle ASM). If the 5 disks are successfully added (CACHED and MEMBER), then go to Step 7.

      su grid user /opt/oracle/oak/bin/stordiag/asm_script.sh 1 6
      

      For example:

      #/opt/oracle/oak/bin/stordiag/asm_script.sh 1 6 | grep CACHED
      .......
      /dev/mapper/SSD_E0_S05_1399652120p1 SSD_E0_S05_1399652120P1 1 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S05_1399652120p2 SSD_E0_S05_1399652120P2 3 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S06_1399645200p1 SSD_E0_S06_1399645200P1 1 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S06_1399645200p2 SSD_E0_S06_1399645200P2 3 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S07_1399646692p1 SSD_E0_S07_1399646692P1 1 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S07_1399646692p2 SSD_E0_S07_1399646692P2 3 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S08_1399649840p1 SSD_E0_S08_1399649840P1 1 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S08_1399649840p2 SSD_E0_S08_1399649840P2 3 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S09_1399649424p1 SSD_E0_S09_1399649424P1 1 NORMAL ONLINE CACHED MEMBER
      /dev/mapper/SSD_E0_S09_1399649424p2 SSD_E0_S09_1399649424P2 3 NORMAL ONLINE CACHED MEMBER

   2. If the disks are not added to Oracle ASM, then add them manually. As grid user, execute the sqlplus '/as sysasm' command on the master node to add the disks to Oracle ASM.

      For a system without Oracle Automatic Storage Management Filter Driver (Oracle ASM Filter Driver) configured, add the Oracle ASM disks as follows:

      
      SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk 
      '/dev/mapper/SSD_E0_S05_1399764284p1' name SSD_E0_S05_1399764284p1, 
      '/dev/mapper/SSD_E0_S06_1399765076p1' name SSD_E0_S06_1399765076p1, 
      '/dev/mapper/SSD_E0_S07_1399765116p1' name SSD_E0_S07_1399765116p1, 
      '/dev/mapper/SSD_E0_S08_1399765484p1' name SSD_E0_S08_1399765484p1, 
      '/dev/mapper/SSD_E0_S09_1399765504p1' name SSD_E0_S09_1399765504p1;  
      
      SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk 
      '/dev/mapper/SSD_E0_S05_1399764284p2' name SSD_E0_S05_1399764284p2, 
      '/dev/mapper/SSD_E0_S06_1399765076p2' name SSD_E0_S06_1399765076p2, 
      '/dev/mapper/SSD_E0_S07_1399765116p2' name SSD_E0_S07_1399765116p2, 
      '/dev/mapper/SSD_E0_S08_1399765484p2' name SSD_E0_S08_1399765484p2, 
      '/dev/mapper/SSD_E0_S09_1399765504p2' name SSD_E0_S09_1399765504p2;  
      
      

      For a system with Oracle Automatic Storage Management Filter Driver (Oracle ASM Filter Driver) configured, add the Oracle ASM disks as follows:

      SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk 
      'AFD:SSD_E0_S05_1399764284P1' name SSD_E0_S05_1399764284p1, 
      'AFD:SSD_E0_S06_1399765076P1' name SSD_E0_S06_1399765076p1, 
      'AFD:SSD_E0_S07_1399765116P1' name SSD_E0_S07_1399765116p1, 
      'AFD:SSD_E0_S08_1399765484P1' name SSD_E0_S08_1399765484p1, 
      'AFD:SSD_E0_S09_1399765504P1' name SSD_E0_S09_1399765504p1;  
      
      SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk 
      'AFD:SSD_E0_S05_1399764284P2' name SSD_E0_S05_1399764284p2, 
      'AFD:SSD_E0_S06_1399765076P2' name SSD_E0_S06_1399765076p2, 
      'AFD:SSD_E0_S07_1399765116P2' name SSD_E0_S07_1399765116p2, 
      'AFD:SSD_E0_S08_1399765484P2' name SSD_E0_S08_1399765484p2, 
      'AFD:SSD_E0_S09_1399765504P2' name SSD_E0_S09_1399765504p2;  
      

6. Use the odaadmcli show validation storage errors command to show hard storage errors.
   Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.
   # odaadmcli show validation storage errors
7. Use the odaadmcli show validation storage failures command to show soft validation errors.
   A typical soft disk error would be an invalid version of the disk firmware.
   # odaadmcli show validation storage failures
8. Confirm that the oak_storage_conf.xml file shows the number of disks added on both nodes, after the addition. For example, if you added 10 disks to the base configuration, then the numberOfDisks option must show 19.

   #cat /opt/oracle/oak/conf/oak_storage_conf.xml
   <!--
   This file is created by the ODA software as part of system provisioning
   based on system provisioning requests. Values of element nodes can be
   changed by OAK in response to storage configuration change operation.
   DO NOT EDIT THIS FILE.
   -->
   <CometConfiguration>
     <OakStorageConfigInfo type="string" dimension="vector" readonly="true" required="true" default="">
       <!-- Number of disks part of OAK -->
       <numberOfDisks>12</numberOfDisks>
       <!-- Number of partitions per disk part of DATA diskgroup in single partition scheme -->
       <!-- All ASM diskgroups in single partition scheme will have same number of partition -->
       <numOfDataDiskPartitionInAsm>1</numOfDataDiskPartitionInAsm>
     </OakStorageConfigInfo>
   </CometConfiguration>

Use the following procedure only if you want to repurpose an existing storage expansion shelf from one Oracle Database Appliance system to another.

1. Install and cable the storage expansion shelf, but do not power on the expansion shelf.

   Caution:

   Review cabling instructions carefully to ensure that you have carried out cabling correctly. Incorrect connections can cause data loss when adding a storage expansion shelf to Oracle Database Appliance with existing databases.

2. Verify that both nodes plus the base storage shelf are up and running. Log into each server node and run the odacli validate-storagetopology command to confirm that the base configuration cabling is correct.

   # odacli validate-storagetopology
    ...
         INFO  : Check if JBOD powered on
    SUCCESS    : JBOD : Powered-on                                          
         INFO  : Check for correct number of EBODS(2 or 4)
    SUCCESS    : EBOD found : 2                                                                                                                                                                     INFO       : Check for overall status of cable validation on Node0
     SUCCESS   : Overall Cable Validation on Node0            
    SUCCESS    : JBOD Nickname set correctly : Oracle Database Appliance - E0

   Run the command to confirm that the two server nodes are properly cabled to the base storage shelf and all disks are online, with a good status, and added to the existing disk groups on both nodes. If there any failures, then fix the cabling before proceeding to the next step.

   Note:

   If the output shows that EBOD found is 2, then you only have the base storage shelf. If EBOD found is 4, then you have a base storage shelf and an expansion shelf.

   Note:

   If the JBOD was configured earlier, then the EBOD found message is displayed. If an unconfigured JBOD is added, then a warning message is displayed.

    odacli validate-storagetopology
    ...
   WARNING : JBOD Nickname is incorrectly set to :

3. Power on the storage expansion shelf.
4. Log in to each server node and run the odacli validate-storagetopology command to validate the storage cabling and confirm that the new storage shelf is recognized.

   
   # odacli validate-storagetopology
   
     INFO    : Check if JBOD powered on
     SUCCESS : 2JBOD : Powered-on                                               
     INFO    : Check for correct number of EBODS(2 or 4)
     SUCCESS : EBOD found : 4                                                   
      ...
      ...
   
      INFO    : Check for overall status of cable validation on Node0
      SUCCESS : Overall Cable Validation on Node0            
      SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
      SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1                 

   Look for the following indicators that both storage shelves are recognized:

   • When there are two shelves, the JBOD (just a bunch of disks) is numbered. For example:

     SUCCESS : 2JBOD : Powered-on

   • When both shelves are recognized, the EBOD found value is 4.

     SUCCESS : EBOD found : 4

   • When the expansion shelf is cabled properly, the nickname is E1. For example:

             SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
             SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1  

   Fix any errors before proceeding.

5. Run the odaadmcli show disk command to ensure that all disks in the expansion shelf are listed, are online, and are in a good state.

   # odaadmcli show disk

   When all disks are online and in a good state, proceed to the next step.
6. Run the odaadmcli show enclosure command to check the health of components in expansion shelf.

   # odaadmcli show enclosure

7. Run the odaadmcli show ismaster command on Node 0 to confirm that Node 0 is the master.

   # odaadmcli show ismaster

8. Run the odaadmcli expand storage command on the master node.

   # odaadmcli expand storage -ndisk 24 -enclosure 1 
   
   Running precheck, it may take a few minutes.
   Precheck passed.
   Check the progress of expansion of storage by running 'odaadmcli show disk'
   Waiting for expansion to finish. It may take several minutes to complete
   depending upon the number of disks being expanded
    
   odaadmcli expand storage command completed.
   Check status of newly expanded disk in OAK by running 'odaadmcli show disk'
   All of them must be ONLINE status. Check status of newly expanded disk in ASM
   by running 'asmcmd lsdsk -p', mount status must be CACHED and header status
   should be MEMBER for all the newly added disks. If the status of newly added
   disks in OAK and ASM are different then expand storage command failed.

   It takes approximately 30 to 40 minutes to add all of the disks to the configuration.
9. Use the odaadmcli show validation storage errors command to show hard storage errors.
   Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.

   # odaadmcli show validation storage errors

10. Use the odaadmcli show validation storage failures command to show soft validation errors.
    A typical soft disk error would be an invalid version of the disk firmware.

    # odaadmcli show validation storage failures

11. Run the odacli describe-component command to verify that all firmware components in the storage expansion are current.

    # odaadmcli describe-component

12. If needed, update the storage shelf and then run the odacli describe-component command to confirm that the firmware is current.

    # odaadmcli update
    # odaadmcli describe-component