2.14.8 Remove Unnecessary Quorum Disks

Quorum disks are required to ensure data availability and integrity only when an Oracle ASM disk group has an insufficient number of failure groups. For Oracle ASM high redundancy disk groups, quorum disks are required when fewer than five failure groups exist. For normal redundancy disk groups, quorum disks are required when fewer than three failure groups exist.

If storage capacity increases to the point where quorum failure groups are no longer required, you can remove the unnecessary quorum disks. Note that removing unnecessary quorum disks is not mandatory. However, to simplify the system configuration and avoid potential misunderstanding and misconfiguration caused by their presence, Oracle recommends cleanly relocating voting files and removing the unnecessary quorum disks.

Use this procedure to check your system and remove unnecessary quorum disks.

Perform the following checks before removing quorum disks:

  1. Connect to your Oracle ASM instance as an ASM administrator and run the following query to determine whether the system contains unnecessary quorum disks: 

    SQL> SELECT dg.name DISK_GROUP_NAME, dg.state, dg.type REDUNDANCY, d.failgroup_type FAILGROUP_TYPE, COUNT(distinct d.failgroup) FAILURE_GROUPS
     FROM v$asm_diskgroup dg JOIN v$asm_disk d USING (group_number)
     GROUP BY dg.name, dg.state, dg.type, d.failgroup_type;     

DISK_GROUP_NAME                STATE       REDUNDANCY FAILGROUP_TYPE  FAILURE_GROUPS
------------------------------ ----------- ---------- --------------- --------------
DATA1                          MOUNTED     HIGH       QUORUM                       2
DATA1                          MOUNTED     HIGH       REGULAR                      7
RECO1                          MOUNTED     HIGH       REGULAR                      7

    The example output shows that all high redundancy disk groups (DATA1 and RECO1) have seven regular failure groups, which is above the threshold where quorum disks are required. Therefore, the quorum failure groups are no longer required and can be removed. The commands in the remainder of this procedure build on this example. Modify the example commands as needed for your environment.

    Note:

    Do not proceed if your system has insufficient regular failure groups and still requires quorum failure groups.

  2. Gather details about the quorum disks.

    For example:

    SQL> SELECT dg.name disk_group, d.failgroup_type, d.name, d.mode_status, d.state, d.header_status
     FROM v$asm_diskgroup dg JOIN v$asm_disk d USING (group_number) 
     WHERE d.failgroup_type = 'QUORUM' ;

DISK_GROUP     FAILGROUP_TYPE  NAME                           MODE_STATUS STATE    HEADER_STATUS
-------------- --------------- ------------------------------ ----------- -------- ------------
DATA1          QUORUM          QD_DATA1_DBNODE01              ONLINE      NORMAL   MEMBER
DATA1          QUORUM          QD_DATA1_DBNODE02              ONLINE      NORMAL   MEMBER

    Note the name of each quorum disk. Also, check that MODE_STATUS=ONLINE, STATE=NORMAL, and that each quorum disk belongs to the corresponding disk group.

    Note:

    If any quorum disk has an issue, stop and contact Oracle Support.

  3. Take note of the quorum disk devices, targets, and configuration details.

    As the root OS user, run the following commands on every database node and note the output: 

    # /opt/oracle.SupportTools/quorumdiskmgr --list --device
    # /opt/oracle.SupportTools/quorumdiskmgr --list --target
    # /opt/oracle.SupportTools/quorumdiskmgr --list --config

    For example:

    [root@dbnode01 ~]#  /opt/oracle.SupportTools/quorumdiskmgr --list --device

Device path: /dev/exadata_quorum/QD_DATA1_DBNODE01
Host name: dbnode01
ASM disk group name: DATA1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATA1_DBNODE02
Host name: dbnode02
ASM disk group name: DATA1
Size: 128 MB

[root@dbnode02 ~]#  /opt/oracle.SupportTools/quorumdiskmgr --list --device

Device path: /dev/exadata_quorum/QD_DATA1_DBNODE01
Host name: dbnode01
ASM disk group name: DATA1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATA1_DBNODE02
Host name: dbnode02
ASM disk group name: DATA1
Size: 128 MB
    [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --target

...

Name: iqn.2015-05.com.example:qd--data1--dbnode01
Host name: dbnode01
ASM disk group name: DATA1
Visible to: iqn.1988-12.com.example:192.168.8.53, iqn.1988-12.com.example:192.168.8.54, iqn.1988-12.com.example:192.168.8.55, iqn.1988-12.com.example:192.168.8.56
Discovered by: 192.168.8.53, 192.168.8.55, 192.168.8.56

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --target

...

Name: iqn.2015-05.com.example:qd--data1--dbnode02
Host name: dbnode02
ASM disk group name: DATA1
Visible to: iqn.1988-12.com.example:192.168.8.53, iqn.1988-12.com.example:192.168.8.54, iqn.1988-12.com.example:192.168.8.55, iqn.1988-12.com.example:192.168.8.56
Discovered by: 192.168.8.53, 192.168.8.55
    [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oracle
Group: dba
ifaces: exadata_re0 exadata_re1
Initiator name: iqn.1988-12.com.example:192.168.8.53

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oracle
Group: dba
ifaces: exadata_re0 exadata_re1
Initiator name: iqn.1988-12.com.example:192.168.8.55

Use the following procedure to remove unnecessary quorum disks:

  1. Check the location of the Oracle Clusterware voting files and move them if necessary.

    As the Oracle Grid Infrastructure (GI) software owner, run crsctl query css votedisk on any GI cluster node.

    For example:

    $ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   08652539d6694f8bbfaddd2036a03f71 (/dev/exadata_quorum/QD_DATA1_DBNODE01) [DATA1]
 2. ONLINE   e98aeb23a2854f3bbf3d5643be65e288 (o/203.0.113.17;203.0.113.18/DATA1_CD_03_celadm05) [DATA1]
 3. ONLINE   f6651f5154e34f74bfbb568ecd649be2 (/dev/exadata_quorum/QD_DATA1_DBNODE02) [DATA1]
 4. ONLINE   fe499a5fe64c4f55bf7b3c7017fa05c4 (o/203.0.113.9;203.0.113.10/DATA1_CD_05_celadm01) [DATA1]
 5. ONLINE   3ba9bea664314f1bbf8a2916eef10f5d (o/203.0.113.15;203.0.113.16/DATA1_CD_07_celadm04) [DATA1]
Located 5 voting disk(s).

    Compare the voting file names with the quorum disk device names that you previously gathered to determine whether any voting file resides on a quorum disk.

    If no voting files reside on quorum disk devices, skip directly to the next step. Otherwise, run crsctl replace votedisk +<DISKGROUP_NAME> to move all voting disks to regular failure groups associated with the specified disk group.

    For example:

    $ crsctl replace votedisk +RECO1

    See also Adding, Deleting, or Migrating Voting Files in Oracle Clusterware Administration and Deployment Guide.

  2. Drop the quorum disks from the disk group.

    Connect to your Oracle ASM instance as an ASM administrator and drop each quorum disk identified above (in prerequisite step 2).

    For example:

    SQL> ALTER DISKGROUP DATA1 DROP QUORUM DISK QD_DATA1_DBNODE01;

Diskgroup altered.

SQL> ALTER DISKGROUP DATA1 DROP QUORUM DISK QD_DATA1_DBNODE02;

Diskgroup altered.

    If a quorum disk is inaccessible, you can use the FORCE keyword. For example: 

    SQL> ALTER DISKGROUP DATA1 DROP QUORUM DISK QD_DATA1_DBNODE01 FORCE;

    Monitor the operation by querying gv$asm_operation, and proceed only when no rows are returned for the disk group.

  3. Remove the quorum disk devices, targets, and configuration metadata.

    Use the information that you gathered above (in prerequisite step 3).

    1. Delete all quorum disk devices across all database nodes.

      On each cluster node, as the root OS user, run the following command: 

      # /opt/oracle.SupportTools/quorumdiskmgr --delete --device --asm-disk-group <DISKGROUP_NAME> --host-name <HOST_NAME>

      You may need to run the command multiple times on each node to cover all of the quorum disk devices across all the database nodes.

      For example:

      [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --device --asm-disk-group DATA1 --host-name dbnode01
[INFO     ] [Success] deleted device /dev/exadata_quorum/QD_DATA1_DBNODE01.

[root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --device --asm-disk-group DATA1 --host-name dbnode02
[INFO     ] [Success] deleted device /dev/exadata_quorum/QD_DATA1_DBNODE02.

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --device --asm-disk-group DATA1 --host-name dbnode01
[INFO     ] [Success] deleted device /dev/exadata_quorum/QD_DATA1_DBNODE01.

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --device --asm-disk-group DATA1 --host-name dbnode02
[INFO     ] [Success] deleted device /dev/exadata_quorum/QD_DATA1_DBNODE02.

      Proceed only when no quorum disk devices are visible on any database nodes (no output is returned from the quorumdiskmgr --list --device command). For example: 

      [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --device

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
    2. Remove the quorum disk targets on all database nodes.

      On each cluster node, as the root OS user, run: 

      # /opt/oracle.SupportTools/quorumdiskmgr --delete --target --asm-disk-group <DISKGROUP_NAME>

      For example:

      [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --target --asm-disk-group DATA1
[INFO     ] [Success] removed target iqn.2015-05.com.oracle:qd--data1--dbnode01
[INFO     ] [Success] removed backstore QD_DATA1_DBNODE01
[INFO     ] [Success] removed logical volume /dev/VGExaDb/LVDbVdDBNODE01DATA1

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --target --asm-disk-group DATA1
[INFO     ] [Success] removed target iqn.2015-05.com.oracle:qd--data1--dbnode02
[INFO     ] [Success] removed backstore QD_DATA1_DBNODE02
[INFO     ] [Success] removed logical volume /dev/VGExaDb/LVDbVdDBNODE02DATA1
    3. Delete the quorum configuration metadata on all of the database nodes.

      On each cluster node, as the root OS user, run: 

      # /opt/oracle.SupportTools/quorumdiskmgr --delete --config

      For example:

      [root@dbnode01 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --config
[INFO     ] [Success] deleted quorum disk configurations

[root@dbnode02 ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --config
[INFO     ] [Success] deleted quorum disk configurations
  4. Update the Oracle ASM disk group discovery string.

    Modify the asm_diskstring parameter to remove the discovery path for quorum disks.

    For example:

    1. Check the asm_diskstring parameter setting: 

      SQL> show parameter asm_diskstring

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------------------------------------------------------------------
asm_diskstring                       string      o/*/DATA1_*, o/*/RECO1_*, /dev/exadata_quorum/QD_DATA*

    2. Take a backup of the ASM server parameter file (spfile) before making any changes.

      SQL> create pfile='/tmp/asm_spfile_bakup.ora' from spfile;

File created.

    3. Update the asm_diskstring parameter, removing the discovery path for quorum disks: 

      SQL> alter system set asm_diskstring='o/*/DATA1_*', 'o/*/RECO1_*' scope=both sid='*';

    4. Verify the updated asm_diskstring parameter setting: 

      SQL> show parameter asm_diskstring

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------------------------------------
asm_diskstring                       string      o/*/DATA1_*, o/*/RECO1_*

Parent topic: Managing Quorum Disks