The global service may be down. Verify that the pool databases are up and the service is started.

The global service may be disabled. Ensure that the pool databases are up and the service is enabled and started.

The GDS pool database may be down. Ensure that the GDS pool databases are up and the service is enabled and started.

The GDS pool database's local listener may be down. Ensure that the GDS pool database local listener is running.

All global service managers may be down. Verify that the global service managers are running.

GSM restart after GSM-upgrade fails with error: GSM-40167: VNCR entry "<hostname>" is not resolvable on GSM host

This issue is most likely due to the presence of short hostnames in the VNCR list causing an issue during GSM start in 23ai. If this issue is encountered, remove invitednode of the two short shard hostname entries and then try to start the GSM.

The user GSMADMIN_INTERNAL is an internal only user, it should never be unlocked or used for any database login. No direct modifications should be made on the Global Data Services schema objects unless directed by Oracle Technical Support.

The GDS catalog database or its listener may be down. Verify that the GDS catalog database and its local listener are running.

You need multiple address lists; the first list should be exclusively regional global service manager listeners, the second list contains global service manager listeners of the buddy region and the third list contains local listeners.

You can always connect through a global service manager while it is up, and only fail over to local listeners when all global service manager listeners are down.

Template:

(DESCRIPTION=
  (FAILOVER=on)
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(local_protocol_address_information))
    (ADDRESS=(local_protocol_address_information))
   )
  (CONNECT_DATA=
   (SERVICE_NAME=global_service_name)
   (REGION=region_name)))

Example:

(DESCRIPTION=
     (FAILOVER=on)
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost1)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost2)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost3)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost4)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost5)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost6)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=server1)(PORT=1521)(PROTOCOL=TCP))
      )
     (CONNECT_DATA=
      (SERVICE_NAME=sales_read_service.dbpoolora.oradbcloud)
      (REGION=WEST)))

The GDS catalog database or its listener may be down. Verify that the GDS catalog database and its local listener are running.

If logging and tracing has been enabled for GDS components, you can find the exact location of a given global service manager's log and trace files using the status gsm command as shown in the following example.

GDSCTL>status gsm
 
Alias                     MYGSM
Version                   19.1.0.0.1
Start Date                22-OCT-2022 14:05:11
Trace Level               support
Listener Log File         /scratch/oracle/diag/gsm/myhost/mygsm/alert/log.xml
Listener Trace File       /scratch/oracle/diag/gsm/myhost/mygsm/trace/ora_1829_
 47542149303936.trc
Endpoint summary          (ADDRESS=(HOST=myhost.com)(PORT=1571)(PROTOCOL=tcp))
GSMOCI Version            0.1.7
Mastership                N
Connected to GDS catalog  Y
Process Id                1833
Number of reconnections   0
Pending tasks.     Total  0
Tasks in  process. Total  0
Regional Mastership       TRUE
Total messages published  34261
Time Zone                 -07:00
Orphaned Buddy Regions:   None
GDS region                east
Network metrics:
   Region: euro RTT:34 Bandwidth:40

In this example myhost is the global service manager host name and mygsm is the name of the global service manager.

Although not strictly a GDS component, the LISTENER log file can be helpful resolving some issues. The text based LISTENER log can be found in /scratch/oracle/diag/gsm/hostname/gsm_name/trace directory. The file is called alert_gsm*.log (for example, alert_gsm1.log)

If logging and tracing have not been enabled for GDS components, follow the steps below to do so:

1. Enable logging for GDSCTL. On the GSM host, edit the $ORACLE_HOME/network/admin/gsm.ora file and add the following: _GDSCTL=(log=ALL)

   This will log any commands run using GDSCTL in the following log file on the GSM host. $ORACLE_HOME/network/admin/GDSCTL.log

2. Enable tracing for GSM processes.

   GDSCTL> set trace_level -gsm <gsm_name> SUPPORT

   This will generate trace files in DIAGNOSTICS_DEST for GSM. For example, $ORACLE_BASE/diag/gsm/<hostname>/<gsm-name>/trace

3. Enable tracing on the catalog database.

   SQL> alter system set events '10798 trace name context forever, level 7';

4. Enable tracing on the GDS pool databases.

   SQL> alter system set events '10798 trace name context forever, level 7';

To disable logging and tracing, follow the steps below:

1. Turn off GSM tracing.

   GDSCTL> set trace_level -gsm <gsm-name> OFF

2. Turn off GDSCTL command logging by editing the gsm.ora file and removing the following line:

   _GDSCTL=(log=ALL)

3. Turn off tracing on the catalog database and GDS pool databases.

   SQL> alter system set events '10798 trace name context forever, level 0';

To effectively troubleshoot Global Data Services, it is important to collect adequate data from GDS and related components. Following the steps below will provide a good data pool for troubleshooting issues:

1. Collecting patch information can be useful for code version-related issues:

   $ cd $ORACLE_HOME/OPatch $ ./opatch lsinventory -detail > lsinventory_info.txt

2. Collect all GDS configuration data. Save the following commands in a script and execute it:

   #!/bin/bash
   # Stop script if any command fails
   set -e
   # Check if gdsctl is available in PATH
   if ! command -v gdsctl &> /dev/null
   then
       echo "gdsctl could not be found. Please make sure it is installed and available in PATH."
       exit 1
   fi
   echo "Starting gdsctl session..."
   # Start the gdsctl session
   gdsctl << EOF
   # List GSMs, GSM status
   echo "GDSCTL COMAND: config"
   config
   echo "GDSCTL COMAND: config gsm -gsm <gsm name>"
   #config gsm -gsm <gsm name>
   echo "GDSCTL COMAND: gsm status"
   gsm status
   # List services
   echo "GDSCTL COMAND: services"
   services
   echo "GDSCTL COMAND: config service -service <service name>"
   #config service -service <service name>
   # List databases
   echo "GDSCTL COMAND: databases"
   databases
   # Network
   echo "GDSCTL COMAND: config vncr"
   config vncr
   # Validate config
   echo "GDSCTL COMAND: validate"
   validate
   # End of gdsctl commands
   EOF
   echo "gdsctl session completed successfully."
   # Exit the script
   exit 0

3. Collect GSM listener status.

   $ lsnrctl status <gsm-name>

If it is necessary to contact Oracle Support, the above data will prove useful. In addition, the following files would also be helpful:

From GSM Host: $ORACLE_HOME/network/admin

From GSM Host: GSM alert log file

From GSM Host: GDSCTL.log

From GSM Host: GSM trace files located in <DIAGNOSTIC_DEST>

From GSM Host: lsinventory, lsinventory_info.txt

From GDS Catalog Host: lsinventory, lsinventory_info.txt