4 GDS Administration

The GDSCTL utility is used to create, manage and monitor a Global Data Services configuration and all of its components. This utility is very similar to the SRVCTL utility used to manage an Oracle Real Application Cluster (Oracle RAC). The following topics explain how to administer your GDS configurations.

4.1 Overview of Global Data Services Administration

Global Data Services is managed by the Global Data Services administrator whose responsibilities include the following tasks:

  • Installing and upgrading the global service manager software

  • Creation and maintenance of the Global Data Services catalog

  • Starting, stopping, and configuring global service managers

  • Creation and administration of Global Data Services regions and pools

  • Management of global services

  • Monitoring of the Global Data Services framework components

Each Global Data Services configuration requires at least one Global Data Services administrator. A small configuration can be administered by a single person who performs all the administrative duties. For a large configuration with many regions and pools it may be necessary to have a group of Global Data Services administrators who share responsibilities. All Global Data Services administrators have privileges to perform all the listed administrative tasks for a given Global Data Services configuration.

An operating system account should exist for the Global Data Services administrator on all computers where global service managers are expected to run. The account user should have privileges to install and run global service manager software. Only Global Data Services administrators should be granted these privileges.

A Global Data Services administrator must also be added as a user to the Global Data Services catalog database and granted the GSMADMIN_ROLE role. The database account for a Global Data Services administrator should be created by a database administrator of the catalog database. The Global Data Services administrator might create this account by himself if he happens to have local database administrator privileges on this database.

If a Global Data Services configuration contains multiple pools, then in addition to Global Data Services administrators who manage the entire configuration, each pool can have one or more Global Data Services pool administrators. Responsibilities of a pool administrator are limited to the administration of a particular pool and include the following tasks:

  • Adding and removing databases in the pool

  • Management of global services in the pool

To perform these tasks a Global Data Services pool administrator must be a user of the Global Data Services catalog database with the appropriate privileges. Creation of the database user for a pool administrator and granting of the privileges is performed automatically when a Global Data Services pool is created with the -USER option. A pool administrator can also be added to a pool after its creation using gdsctl modify gdspool command. A Global Data Services administrator always has privileges to administer any pool in the database configuration.

All administrative operations should be performed using the appropriate GDSCTL commands. Execution of the most GDSCTL commands requires access to the Global Data Services catalog. For such commands, credentials for the catalog database must be specified using the appropriate command options.

Many administrative operations, such as adding a database to a Global Data Services pool, or enabling a global service, require making changes not only to the Global Data Services catalog, but also to databases in the Global Data Services configuration. The generic workflow for such commands is as follows:

  • GDSCTL connects to the catalog database with credentials provided by the administrator and makes appropriate changes to the catalog.

  • The catalog database notifies all global service managers in the Global Data Services configuration about the changes. The notification is sent using an Oracle Net Services connection that each global service manager maintains with the catalog database.

  • After receiving the notification one of the global service managers connects to the configuration databases that need to be configured and makes the appropriate changes.

To support this workflow a global service manager should be able to connect to the catalog and configuration databases. The connection to the catalog database is established using GSMCATUSER account, which is created by default on any Oracle database during database installation. The account must be unlocked by the database administrator of the catalog database and its password given to the Global Data Services administrator. Whenever a new global service manager is added to the GDS configuration, the Global Data Services administrator has to specify the password for the GSMCATUSER account. The password is then encrypted and stored in the global service manager wallet for future use by the global service manager.

The global service manager connects to the pool databases using the GSMUSER account, which also exists by default on any Oracle database. The account is locked after the database installation. It should be unlocked by the local database administrator before the database can be added to a Global Data Services pool. The password for the GSMUSER account is given to the pool or Global Data Services administrator who adds the database to a Global Data Services pool and must be specified in the gdsctl add database command. The password is stored in the Global Data Services catalog for future use by all global service managers.

4.2 Managing the GDS Stack

This section describes the startup and shutdown of components in the global data services framework.

4.2.1 Starting Up the GDS Stack

The following is the recommended startup sequence of the GDS stack:

  • Start the global data services catalog database and local listener.

  • Start the global service managers.

  • Start the GDS pool databases and local listeners.

  • Start the global services.

  • Start the application tier and the clients.

4.2.2 Shutting Down the GDS Stack

The following is the recommended shutdown sequence of the GDS stack:

  • Shut down the application tier and the clients.

  • Stop the global services.

  • Shut down the GDS pool databases and local listeners.

  • Stop the global service managers.

  • Stop the global data services catalog database and the local listener.

4.3 Monitoring the GDS Environment

Oracle Global Data Services (GDS) implements the Oracle Database service model across a set of replicated databases known as a Global Data Services configuration.

To monitor the global services configuration:

GDSCTL> services
Service "sales_sb.sales.oradbcloud" has 2 instance(s). Affinity: ANYWHERE
Instance "sales%1", name: "mts1", db: "mts", region: "slc", status: ready.
Instance "sales%2", name: "mts2", db: "mts", region: "slc", status: ready.

Monitoring member databases:

GDSCTL> databases
Database: "mts" Registered: Y State: Ok ONS: Y. Role: PRIMARY
Instances: 1 Region: slc
Registered instances:
sales%1

To monitor the global service manager configuration:

GDSCTL> status
Alias GSM1
Version 19.17.0.3.0
Start Date 13-APR-2023 09:40:59
Trace Level off
Listener Log File
/u01/app/oracle/diag/gsm/hostname/gsm1/alert/log.xml
Listener Trace File
/u01/app/oracle/diag/gsm/hostname/gsm1/trace/ora_64863_139739749930432.trc
Endpoint summary
(ADDRESS=(HOST=hostname.example.com)(PORT=1522)(PROTOCOL=tcp))
GSMOCI Version 0.6.11
Mastership Y
Connected to GDS catalog Y
Process Id 64883
Number of reconnections 0
Pending tasks. Total 0
Tasks in process. Total 0
Regional Mastership TRUE
Total messages published 0
Time Zone -04:00
Orphaned Buddy Regions: None
GDS region regionora

4.4 Managing Global Data Services

This section describes the administration tasks associated with Global Data Services.

4.4.1 Managing GDS Regions

What You Need to Know About Adding a Global Data Services Region

If you require only one Global Data Services region, then you do not need to add a region using these instructions. A default Global Data Services region, REGIONORA, is created for you when you create the Global Data Services catalog.

For example:

GDSCTL> add region –region west,east

The preceding example adds two regions, east and west, to the Global Data Services framework.

A Global Data Services region should have a name that is unique within the corresponding Global Data Services configuration. If no name is specified at the first region creation time, the default name, oraregion, is given to the region. The region name can be up to 30 characters long and can be any valid identifier - an alphabetical character followed by zero or more alphanumeric ASCII characters or '_'.

To modify the configuration parameters for an existing region, use the modify region command: 

GDSCTL> modify region -region west -buddy east

Where -buddy indicates a "buddy" region.

To remove a specified region(s) from the global service management framework, use the remove region command: 

GDSCTL> GDSCTL> remove region -region south

4.4.2 Managing GDS Pools

Adding a Global Data Services Pool

Ensure that you are connected to the Global Data Services catalog and add a pool, administered by a specific user, as follows: 

GDSCTL> add gdspool -gdspool database_pool_list [-users user_list]

If you require only one Global Data Services pool, then you do not need to add one using this example. A default Global Data Services pool, DBPOOLORA, is created for you when you create the Global Data Services catalog.

The Global Data Services administrator has permissions to run GDSCTL commands to manage a Global Data Services pool and, if there is only a single pool, then the Global Data Services administrator also administers the pool.

If you specify a user when you run the gdsctl add gdspool command, then the local DBA where the Global Data Services catalog resides must first add the user to the catalog database.

Large database clouds can require multiple Global Data Services pools that are managed by different administrators.

For example:

GDSCTL> add gdspool -gdspool hr -users rjones

The preceding example adds a Global Data Services pool called hr, and adds the user rjones, who is assigned the privileges to administer the hr pool. The privileges enable the pool administrator to add databases to the pool and manage global services on the databases in the pool.

A Global Data Services pool must have a unique name within its GDS configuration. If you do not specify a name for the pool when you create it, then the name defaults to oradbpool. The pool name can be up to 30 bytes long and can be any valid identifier (an alphabetical character followed by zero or more alphanumeric ASCII characters or the underscore (_)).

Use the modify gdspool command to modify the configuration parameters of GDS pools: 

GDSCTL> modify gdspool -gdspool hr -adduser psmith

Use the remove gdspool command to remove a GDS pool(s) from the current configuration: 

GDSCTL> remove gdspool -gdspool tempreaders,myfarm

4.4.3 Managing Member Databases

To provide global services, a database must be added to a Global Data Services pool.

Before adding a database to a pool, the database administrator should unlock the GSMUSER account and give the password to the Global Data Services pool administrator, as shown in the following example: 

ALTER USER gsmuser ACCOUNT UNLOCK;
ALTER USER gsmuser IDENTIFIED BY password;

To be part of a Global Data Services pool, a database must use a server parameter file (SPFILE). An Oracle RAC database should also have SCAN set up.

To add a database:

  1. Connect to the Global Data Services catalog using the Global Data Services pool or Global Data Services administrator credentials, for example:
    GDSCTL> connect rjones@catalog
  2. Run the gdsctl add database command:
    GDSCTL>add database -connect edc007:1521/db14.east.example.com -region east
 -gdspool hr

    In this example edc007:1521/db14.east.example.com is the connect identifier of the database, and then you are prompted for the GSMUSER account password on this database.

Note:

If the pool already contains databases and there are global services associated with the pool, then the services are automatically created on the new database.

Use the modify database command to modify the configuration parameters of the databases in a GDS pool, such as region, connect identifier, global service manager password, SCAN address, and ONS port: 

GDSCTL> modify database -database db1,db3 -region east

Use the remove database command to remove databases from a GDS pool: 

GDSCTL> remove database -database db1 -gdspool pool1
4.4.3.1 Valid Node Checking for Registration

The valid node checking for registration (VNCR) feature provides the ability to configure and dynamically update a set of IP addresses, host names, or subnets from which registration requests are allowed by the global service manager. Database instance registration with a global service manager succeeds only when the request originates from a valid node.

By default, the Global Data Services framework automatically adds a VNCR entry for the host on which a remote database is running each time the gdsctl add database command is run. The automation (called auto-VNCR) requires that the host name entry exists in either the local hosts file or in the name server. If the remote host is identified by a different name on any of the nodes on which the global service manager runs, then the Global Data Services administrator must manually add VNCR entry to the Global Data Services catalog by running the gdsctl add invitednode command.

See Also:

add invitednode (add invitedsubnet) for complete usage information

4.4.3.2 Adding Oracle Data Guard Broker Managed Databases to a Database Pool

When you include an Oracle Data Guard broker configuration in a Global Data Services configuration, you manage the broker configuration as one unit. Only an entire Oracle Data Guard broker configuration can be added to (or deleted from) a database pool. A configuration cannot span multiple pools. An attempt to add or remove an individual database to or from a pool that belongs to a broker configuration results in an error.

The only way to add a database to the pool is to add the database to the broker configuration (using the DGMGRL utility). Adding a database to the broker configuration causes its automatic addition to the database pool to which this configuration belongs. Removing a database from a broker configuration causes its removal from the pool that contains the configuration. This is the only way to remove a database from a pool that contains a broker configuration.

Also, note the following limitations:

  • The set of databases in a database pool can be either:

    • The set of databases that belong to a single broker configuration

    • A set of databases that belong to no broker configuration

    You can add a broker configuration only to an empty database pool and, if a pool already contains a broker configuration, then, to add a database to a database pool, you must add the database to the broker configuration contained in the database pool.

  • Role-based global services are supported only for database pools that contain a broker configuration.

See Also:

Oracle Data Guard Broker for more information about the DGMGRL utility

4.5 Managing Global Services

This section describes the administration tasks associated with global services. It contains the following topics:

4.5.1 Creating a Global Service

A global service is created by execution of the add service command. This command associates the global service with a Global Data Services pool and stores attributes of the service in the Global Data Services catalog. If databases are specified using the —preferred or —available options, the service is created on those specified databases. If the —preferred_all option is used, the service is created on all databases in the Global Data Services pool. For example: 

GDSCTL>add service -service sales_sb -preferred_all -gdspool sales -
notification TRUE

A service that already exists in a Global Data Services pool is also automatically created on a database in the following cases:

  • The service is modified to add a database that is part of the pool.

  • The service has the —preferred_all attribute and a new database is added to the pool.

If this is an Oracle RAC database being added with multiple instances, then you must modify the service to add the database instances.

GDSCTL>modify service -gdspool sales -service sales_sb -database mts -
add_instances -preferred mts1,mts2
GDSCTL>modify service -gdspool sales -service sales_sb -database stm -
add_instances -preferred stm1,stm2
GDSCTL>start service -service sales_sb -gdspool sales

See Also:

add service

4.5.2 Starting a Global Service

The gdsctl start service command is used to start an existing service on the Global Data Services pool databases. 

GDSCTL>start service -service emp_report1 -gdspool hr

If the -role parameter is specified for the service, the service only starts on the databases in which the role matches the specified value. If the -lag parameter is specified for the service, the service only starts on the databases for which replication lag does not exceed the specified value. Unless -preferred_all is specified for the service, the service only starts on the databases that are listed as preferred for the service.

See Also:

start service

modify service

A global service is automatically enabled immediately after it has been created. The term enabled means that the service can be started on a database if the database is eligible for running the service, namely, when the following conditions are met:

  • The database is open and registered with a global service manager.

  • The service has not been disabled on that database.

  • The database role matches the role attribute of the service.

  • The replication lag on the database does not exceed the maximum value specified for the service.

  • The service has not reached its cardinality defined by the number of preferred databases.

  • No other database in the pool is a better candidate for starting the service, for example, the service can be started on an available database only if there is no eligible preferred database.

A newly created global service never gets started automatically until the start service command is executed by the user. This gives the pool administrator control over the initial service startup which may be important in the case when multiple services are being added to the pool and a certain sequence of service startups is required.

A service with the automatic management policy (the default option) must be initially started by executing the start service command without the -database option. This command not only starts the service on all eligible databases in the pool, but also enables the automatic service startup in the following cases:

  • After the service is automatically created on a database that is eligible to run it. (The two cases of automatic service creation are listed in the previous section.)

  • A database that was down gets restarted and is eligible for the service.

  • A database becomes eligible to run the service. This can happen, for example, because the replication lag on a database has decreased to an acceptable value, or the service cardinality has been increased by the user.

The start service command with -database option can be used to start a service with the automatic management policy on particular databases if the service was shut down there by the stop service command described in Stopping a Global Service.

A service with the manual policy must be started manually on each individual database, including when a database gets restarted or becomes eligible to run the service. When executed against a service with the manual policy, the start service command without the -database option starts the service on all eligible databases that are currently present in the pool. If used with the -database option, the command starts the service only on the specified databases, if they are eligible to run it.

Note:

The cases of automatic service startup listed in this section only describe what happens when the start service command is executed against a service with the automatic management policy. They do not include cases when a service is started automatically on a database because of a failover from another database. Service failover is not associated with the start service command, and its behavior is the same for services with automatic and manual management policy.

4.5.3 Stopping a Global Service

A global service running on databases in a Global Data Services pool can be shut down by the stop service command. If the stop service command is executed with the -database option, then the service is stopped on the specified databases; otherwise it is stopped on all databases in the pool. For example: 

GDSCTL>stop service -gdspool readerfarm -service sales_report

Note:

A stopped service with the automatic management policy is restarted if the database where it was running gets restarted and is eligible to run the service. Also, stopping a service with the automatic management policy on all databases in a Global Data Services pool does not prevent the automatic service startup on a new database when the service is created there. To completely disable the automatic startup of a service, its management policy should be changed to manual.

When the service is stopped by the user, the Global Data Services framework considers that database to be temporarily unavailable for this service. Stopping a global service does not cause a service failover event; the service cardinality is temporarily decreased and the global service manager does not attempt to start the service on another database in the pool.

However, a database with a stopped service is still considered a failover target for this service; when the service fails on another database, it can be started on this database if it is eligible to run the service. After the service failover to a database, the service on that database is no longer considered to be stopped by the user.

A stopped service can be manually restarted by executing the start service command.

See Also:

stop service

4.5.4 Enabling a Global Service

A disabled global service can be reenabled on a database by executing enable service command. If the service management policy is AUTOMATIC and the database is eligible for running the service, it is started automatically after being enabled. A service with the MANUAL management policy must be started manually. A database can become a failover target after a service is enabled there.

For example to enable the service G_SALES_REPORT on the database DB1 in the database pool READERFARM:

GDSCTL> enable service -gdspool readerfarm -service g_sales_report -database db1

See Also:

add service

4.5.5 Disabling a Global Service

A global service can be disabled on a database or a set of databases by executing the disable service command. A disabled service cannot be started until it is reenabled. This includes the service failover from another database; a database with the disabled service is never considered a failover target.

A service has to be stopped to be disabled. An error is returned if disable service is executed against a database where the service is running.

To disable and stop the service G_SALES_REPORT on all databases in the database pool READERFARM:

GDSCTL> disable service -gdspool readerfarm -service g_sales_report -database db1

See Also:

enable service

4.5.6 Modifying Global Service Attributes

The modify service command is used to modify global service attributes. In addition to specifying service properties (such as role, maximum lag, load balancing method, and so on) service attributes define on which databases the service should run. Therefore modify service can be used to add a database to a service, remove it from a service, or move a service from one database to another. As the result of the command execution, a service may be created, deleted, started, or stopped on one or more databases in a Global Data Services pool.

Most global service attributes are specified at the service creation time in the add service command and only need to be modified when some changes have to be made. However, a few service attributes related to Oracle RAC databases, must be set by executing the modify service command right after the add service command has been executed. These attributes include the name of the server pool, instance cardinality (UNIFORM/) and some other parameters that are specific to particular Oracle RAC databases. Such attributes cannot be set by the add service command because this command is only used to specify attributes that have the same values for all databases in a Global Data Services pool.

See Also:

modify service

4.5.7 Deleting a Global Service

The remove service command deletes a global service from the Global Data Services pool by removing it from the Global Data Services catalog and all databases where it was created. A service should be stopped before being deleted.

See Also:

remove service

4.5.8 Adding a Service to a Global Data Services Pool

The gdsctl add service command is used to create a service on the Global Data Services pool databases. A simple example of the command is as follows: 

GDSCTL> add service -gdspool hr -service emp_report1 -preferred_all

In this example emp_report1 is the service name and the -preferred_all option means that the service should normally run on all of the databases in the pool.

The service name specified in the 'add service' command can be domain qualified (for example, sales.example.com) or not (for example, sales). If the specified name is not domain qualified, the service is created with the default domain name "<GDS_pool_name>.<GDS_configuration_name>", however the shorter non-domain qualified name can be used with subsequent GDSCTL commands to manage the service. If the specified name is domain qualified, the full domain qualified service name must be used in all subsequent GDSCTL commands used to manage the service.

For Oracle RAC-enabled pool databases, after the service has been added, run GDSCTL modify service to specify which Oracle RAC instance a given global service should run on, as shown in the following example. 

GDSCTL> modify service -service emp_report1 -gdspool hr - database db14
 -modify_instances -preferred db14_n1, db14_n2

A global service name must be unique within a GDS pool and when qualified by domain, must also be unique within a GDS configuration. A global service cannot be created at a database if a local or global service with the same name already exists at that database.

A global service name can contain alphanumeric characters, "_' and '.'. The first character must be alphanumeric. The maximum length of a global service name is 64 characters. The maximum length of a domain qualified global service name is 250 characters.

An Oracle Net connect descriptor used to connect to a global service must contain a domain qualified service name

See Also:

add service and modify service for complete usage information

4.5.9 Global Data Services Failover Across Regions Flow

  1. The administrator has failed over or switched the production database to the secondary site. This is automatic if you are using Data Guard fast-start failover.
  2. The administrator starts the middle-tier application servers on the secondary site if they are not already running.
  3. The wide-area traffic manager selection of the secondary site can be automatic in the case of an entire site failure. The wide-area traffic manager at the secondary site returns the virtual IP address of a load balancer at the secondary site, and clients are directed automatically on the subsequent reconnect. In this scenario, the site failover is accomplished by an automatic domain name system (DNS) failover.
  4. Alternatively, a DNS administrator can manually change the wide-area traffic manager selection to the secondary site for the entire site or specific applications.

    The following is an example of a manual DNS failover:

    • Change the DNS to point to the secondary site load balancer: The primary (primary) DNS server is updated with the new zone information, and the change is announced with DNS NOTIFY.
    • The secondary DNS servers are notified of the zone update with a DNS NOTIFY announcement, and the secondary DNS servers pull the new zone information.
    • Clear affected records from caching DNS servers.

      A caching DNS server is used primarily for performance and fast response. The caching server obtains information from an authoritative DNS server in response to a host query and then saves (caches) the data locally. On a second or subsequent request for the same data, the caching DNS server responds with its locally stored data (the cache) until the response's time-to-live (TTL) value expires. At this time, the server refreshes the data from the zone master. If the DNS record is changed on the primary DNS server, then the caching DNS server does not pick up the change for cached records until TTL expires. Flushing the cache forces the caching DNS server to go to an authoritative DNS server again for the updated DNS information.

    • Flush the cache if the DNS server being used supports such a capability. The following is the flush capability of standard DNS BIND versions:
      • BIND 9.3.0: The command rndc flushname name flushes individual entries from the cache.
      • BIND 9.2.0 and 9.2.1: The cache can be flushed with the command rndc flush.
      • BIND 8 and BIND 9 up to 9.1.3: Restarting the named server clears the cache.
  5. Refresh local DNS service caching.

    Some operating systems might cache DNS information locally in the local name service cache. If so, this cache must also be cleared to recognize DNS updates quickly.

  6. The secondary site load balancer directs traffic to the secondary site middle-tier application server.

4.5.10 Graceful Application Switchover

Database services are used to manage workloads during the planned outage properly. Services must be properly created, and the application must obtain connections from a service.

These recommendations assume using a FAN-aware connection pool, such as Oracle Universal Connection Pool (UCP) to gracefully drain connections without application interruption from a service that is stopped. Your applications can use other connection types that don't support FAN-aware connection pools or have long-running transactions. Ideally, these applications will be disconnected before the maintenance window.

The recommendations below describe how to disconnect some sessions when their transaction ends in a timely manner or, ultimately, when the instance is shut down for maintenance.

The recommended and validated approach to understanding and optimizing your application's connection configuration is provided in the following sections; certain applications may have specific guidelines to follow.

Understanding Your Application's Use of Connections

Understanding how your application obtains and releases its connections is critical to determining whether it can gracefully switch to other instances in the cluster.

Find the following information about your application:

  • What was the workload during the planned outage (OLTP/short or batch/long transactions)?

    • Short transactions using a connection pool such as UCP or ODP.NET can be quiesced rapidly.

    • Long transactions need additional time to quiesce or must have batch jobs stopped or queued at an appropriate time in advance.

  • What type of connection was obtained: Java, OCI, ODP with C#, or ODP with OCI)?

    • UCP, ICC, ODP.NET, and OCI session pools use Fast Application Notification (FAN) to drain the pool quickly; other connections require waiting until the connection is closed (or termination if the application allows)

  • What is the amount of time to wait for the connection pool to quiesce before stopping the service?

    • Useful to know the proper amount of time is given before disconnection is performed

  • Can the application handle disconnection after the transaction completes (applies to batch workloads)?

    • If the application can't handle being disconnected gracefully, it must be stopped before the planned maintenance, or Application Continuity might be an option to avoid interruption.

Services and Application Configuration Best Practices

You must have properly configured services and application attributes to perform a graceful switchover successfully. See My Oracle Support Doc ID 1617163.1 for a matrix of validated drivers and applications clients.

Note:

You must test your configuration to ensure that it is set up and performs switchover properly before relying on it for a production system.

4.6 GDS Patching and Upgrading

What You Need to Know About Upgrading Global Data Services

There are four components that comprise the distributed Global Data Services infrastructure, and each component may reside in a separate installation and may be upgraded independently using the usual upgrade procedure; however, there are certain rules about component versioning that should be followed. The components and rules are as follows:

  • Catalog database: The catalog database is the central repository for the GDS metadata; it is a standard Oracle Database installation. The version of the catalog database must always be greater than or equal to the version of any GDSCTL session that connects to it, and exactly equal to the version of any global service manager server that connects to it, with one exception: to ease migration, the catalog may temporarily have a version greater than some global service manager servers that are connected to it, until any change is made to the catalog, at which time any connected global service manager that is not at the same version will be disconnected, and any stopped global service manager that is not at the same version will not be allowed to connect.

  • GDSCTL client: This component is run in an ad-hoc manner from a terminal session on any system that contains a global service manager installation. The version of the GDSCTL client is the version of the global service manager installation that it runs from.

  • Global service manager server: The version of the global service manager server is the version of the global service manager installation from which the server runs. A global service manager server cannot connect to any catalog database that is at a lower version than itself. A global service manager server cannot connect to any catalog database that is at a higher version than itself in which changes have already been made to the catalog at that higher version.

  • Pool database: A pool database is any database added to a GDS pool which runs a global service. You may upgrade or downgrade pool databases at any time.

Given these rules, it is possible to perform a rolling upgrade of the distributed GDS infrastructure with zero service downtime.

Note:

For general information regarding upgrading and patching see: Oracle Database Upgrade Guide.

Note:

For Oracle Database Proactive Patch Information, see Oracle Support (MOS) Note 2521164.1

4.6.1 Upgrading Global Data Services

The advised order of upgrade is:

  1. Upgrade the catalog database. For best results this should be done using a rolling database upgrade; however, global services will remain available during the upgrade if the catalog is unavailable, although service failover will not occur.
  2. Upgrade global service manager installations that are used to run GDSCTL clients, which do not also run a global service manager server (if any).

    Note:

    Global service manager upgrades should be done in-place; however, an in-place upgrade will cause erroneous error messages unless permissions on the following files for the following platforms are updated to 755:

    On Linux/Solaris64/Solaris Sparc64:

    $ORACLE_HOME/QOpatch/qopiprep.bat

    $ORACLE_HOME/jdk/bin/jcontrol

    $ORACLE_HOME/jdk/jre/bin/jcontrol

    On AIX:

    $ORACLE_HOME/QOpatch/qopiprep.bat

    $ORACLE_HOME/jdk/jre/bin/classic/libjvm.a

    $ORACLE_HOME/jdk/bin/policytool

    On HPI:

    $ORACLE_HOME/jdk/jre/lib/IA64N/server/Xusage.txt

    $ORACLE_HOME/jdk/jre/bin/jcontrol

    $ORACLE_HOME/QOpatch/qopiprep.bat

    On Windows, no error messages are expected.

  3. Stop, upgrade, and restart all global service manager servers one-at-a-time. In order to ensure zero downtime, at least one global service manager server should always be running. Global service manager servers at an earlier version than the catalog will continue to operate fully until catalog changes are made.
  4. Upgrade pool databases in any order, either before or after the global service manager and catalog database upgrades, at the discretion of the pool database administrator.

4.6.2 GSM Out-of-Place Update/Patching Examples

This example shows the steps required to apply the 19.18 Database Realease Update (DBRU) to a 19.3 Oracle environment. The following assumptions are made:
  • A 19.3.0.0.0 GDS Catalog database exists on Host A
  • Two 19.3.0.0.0 GSMs (GSM1 and GSM2) exist on Host B. GSM1 is installed in ORACLE_HOME1 and GSM2 is installed in ORACLE_HOME2
  • A pair of 19.3.0.0.0 Oracle pool databases exist on Host C and Host D respectively.

19.18.0.0.0 DBRU on GDS catalog, two GSMs & two Pool Databases

  1. Stop the GDS catalog database and apply 19.18.0.0.0DBRU then start the GDS catalog database.
    $ORACLE_HOME/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.
    SQL> select PATCH_ID, PATCH_UID, INSTALL_ID, STATUS, ACTION, DESCRIPTION from DBA_REGISTRY_SQLPATCH;

  PATCH_ID  PATCH_UID INSTALL_ID STATUS                    ACTION
---------- ---------- ---------- ------------------------- ---------------
DESCRIPTION
--------------------------------------------------------------------------------
  29517242   22862832          1 SUCCESS                   APPLY
Database Release Update : 19.3.0.0.190416 (29517242)

  34765931   25098466          2 SUCCESS                   APPLY
DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)


SQL>
  2. Verify that the GSM setup is working properly before proceeding (now that the GDS catalog is at 19.18 and the GSMs and pool databases are at version 19.3).
  3. Next, stop GSM1, making GSM2 the new master. Apply the 19.18.0.0.0 DBRU on GSM1.
    $ORACLE_HOME1/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

OPatch succeeded.
    GDSCTL> config

Regions
------------------------
east                          
west                          

GSMs
------------------------
gsm1                          
gsm2                          

GDS pools
------------------------
sdbpool                       

Databases
------------------------
cloud                         
clouddb                       

Services
------------------------
srv1                          

GDSCTL pending requests
------------------------
Command                       Object                        Status                        
-------                       ------                        ------                        

Global properties
------------------------
Name: orasampl
Master GSM: gsm2
DDL sequence #: 0


GDSCTL>
  4. Next, start GSM1 and stop GSM2 (making GSM1 the new master) and apply the 19.18.0.0.0 DBRU on GSM2 and then start GSM2.
    $ORACLE_HOME2/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

OPatch succeeded.


GDSCTL> config

Regions
------------------------
east                          
west                          

GSMs
------------------------
gsm1                          
gsm2                          

GDS pools
------------------------
sdbpool                       

Databases
------------------------
cloud                         
clouddb                       

Services
------------------------
srv1                          

GDSCTL pending requests
------------------------
Command                       Object                        Status                        
-------                       ------                        ------                        

Global properties
------------------------
Name: orasampl
Master GSM: gsm1
DDL sequence #: 0

GDSCTL>
  5. Verify that the GSM environment works properly, now that the GDS catalog & GSM versions are at 19.18 and pool database versions are still at 19.3).
  6. Stop the first pool database and apply the 19.18.0.0.0 DBRU. When complete, start the database.
    $ORACLE_HOME/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.

SQL> select PATCH_ID, PATCH_UID, INSTALL_ID, STATUS, ACTION, DESCRIPTION from DBA_REGISTRY_SQLPATCH;

  PATCH_ID  PATCH_UID INSTALL_ID STATUS                    ACTION
---------- ---------- ---------- ------------------------- ---------------
DESCRIPTION
--------------------------------------------------------------------------------
  29517242   22862832          1 SUCCESS                   APPLY
Database Release Update : 19.3.0.0.190416 (29517242)

  34765931   25098466          2 SUCCESS                   APPLY
DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

SQL>
  7. Stop the second pool database and apply the 19.18.0.0.0 DBRU then restart the database. 
    $ORACLE_HOME/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.

SQL> select PATCH_ID, PATCH_UID, INSTALL_ID, STATUS, ACTION, DESCRIPTION from DBA_REGISTRY_SQLPATCH;

  PATCH_ID  PATCH_UID INSTALL_ID STATUS                    ACTION
---------- ---------- ---------- ------------------------- ---------------
DESCRIPTION
--------------------------------------------------------------------------------
  29517242   22862832          1 SUCCESS                   APPLY
Database Release Update : 19.3.0.0.190416 (29517242)

  34765931   25098466          2 SUCCESS                   APPLY
DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

SQL>
  8. Verify that the GSM setup works correctly now that the GDS catalog, GSMs & pool database versions are at 19.18.0.0.0
    GDSCTL> config

Regions
------------------------
east                          
west                          

GSMs
------------------------
gsm1                          
gsm2                          

GDS pools
------------------------
sdbpool                       

Databases
------------------------
cloud                         
clouddb                       

Services
------------------------
srv1                          

GDSCTL pending requests
------------------------
Command                       Object                        Status                        
-------                       ------                        ------                        

Global properties
------------------------
Name: orasampl
Master GSM: gsm1
DDL sequence #: 0


GDSCTL> 


GDSCTL> databases;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: N Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: N Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL> 


GDSCTL> status database;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: N Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: N Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL> 


GDSCTL> status service;
Service "srv1.sdbpool.orasampl" has 1 instance(s). Affinity: ANYWHERE
   Instance "sdbpool%1", name: "cloud", db: "cloud", region: "east", status: ready.

GDSCTL>
4.6.2.1 GSM_HOME to 19.18.0.0.0 DBRU, Move Existing GSM to New Home on Same Host
  1. Install 19.3.0.0.0 GSM (GSM3) in ORACLE_HOME3 and apply 19.18.0.0.0 DBRU.
    $ORACLE_HOME3/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

OPatch succeeded.
  2. Copy gsm.ora, tnsnames.ora and the gsmwallet directory from the old $TNS_ADMIN folder to the new one.
  3. Stop GSM1 from the old GSM1 home.
    GDSCTL> stop gsm -gsm gsm1;
GSM is stopped successfully
GDSCTL>
  4. Change the WALLET_LOCATION directory to point the new GSM_HOME under gsm.ora.
  5. Start GSM3 from new GSM3 home
    GDSCTL> start gsm -gsm gsm1;
GSM is started successfully
GDSCTL>
  6. Execute modify gsm -gsm <gsm name> from the new home.
    GDSCTL> modify gsm -gsm gsm1
GSM modified
GDSCTL>
  7. Install 19.3.0.0.0 GSM4 on ORACLE_HOME4 and apply 19.18.0.0.0 DBRU.
    $ORACLE_HOME4/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
OPatch succeeded.
  8. Copy gsm.ora, tnsnames.ora and the gsmwallet directory from the old $TNS_ADMIN folder to new one.
  9. Stop GSM2 from the old GSM2 home.
    GDSCTL> stop gsm -gsm gsm2;
GSM is stopped successfully
GDSCTL>
  10. Change the WALLET_LOCATION directory to point to the new GSM_HOME under gsm.ora.
  11. Start GSM4 from the new GSM4 home.
    GDSCTL> start gsm -gsm gsm2;
GSM is started successfully
GDSCTL>
  12. Execute modify gsm -gsm <gsm name> from the new home.
    GDSCTL> modify gsm -gsm gsm2
GSM modified
GDSCTL>
  13. Verify the new GSM environment.
    GDSCTL> config

Regions
------------------------
east                          
west                          

GSMs
------------------------
gsm1                          
gsm2                          

GDS pools
------------------------
sdbpool                       

Databases
------------------------
cloud                         
clouddb                       

Services
------------------------
srv1                          

GDSCTL pending requests
------------------------
Command                       Object                        Status                        
-------                       ------                        ------                        

Global properties
------------------------
Name: orasubbu
Master GSM: gsm1
DDL sequence #: 0


GDSCTL> 


GDSCTL> databases;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: N Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: N Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL>


GDSCTL> status database;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: N Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: N Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL> 


GDSCTL> status service;
Service "srv1.sdbpool.orasubbu" has 1 instance(s). Affinity: ANYWHERE
   Instance "sdbpool%1", name: "cloud", db: "cloud", region: "east", status: ready.

GDSCTL>
4.6.2.2 GSM_HOME to 19.18.0.0.0 DBRU on a Different Host
  1. Install 19.3.0.0.0 GSM1 on ORACLE_HOME1 and apply 19.18.0.0.0 DBRU.
    $ORACLE_HOME1/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

OPatch succeeded.
  2. Copy gsm.ora, tnsnames.ora and the gsmwallet directory from source host to target host ($GSM_HOME/network/admin).
  3. Stop GSM1 on the source host.
  4. Modify the gsm.ora file with target host and target host wallet directory and modify the target host in the tnsnames.ora file for the GSM1 entry.
  5. Modify the GSM with endpoint entry and verify that config gsm points to the correct target host details. For example:
    GDSCTL> modify gsm -gsm gsm1 -endpoint (ADDRESS=(HOST=myhost.example.com)(PORT=1587)(PROTOCOL=tcp))
GSM modified
GDSCTL> 


GDSCTL> config gsm
Name      Region    ENDPOINT                                                    
----      ------    --------                                                    
gsm1      east      (address=(host=myhost.example.com)(port=1587)(protocol=tcp))                       
gsm2      west      (ADDRESS=(HOST=myhost.example.com)(PORT=1787)(PROTOCOL=tcp))                       

GDSCTL>
  6. Start GSM1 from the new GSM1 home.
    GDSCTL> start gsm -gsm gsm1;
GSM is started successfully
GDSCTL>
  7. Execute the modify gsm -gsm <gsm name> -pwd <GSMCATUSER password> command like the example below:
    GDSCTL> modify gsm -gsm gsm1 -pwd <GSMCATUSER secret_password>
GSM modified
GDSCTL>
Perform the following steps on GSM2.
  1. Install 19.3.0.0.0 GSM2 on ORACLE_HOME2 and apply the 19.18.0.0.0 DBRU.
    /$ORACLE_HOME2/OPatch/opatch lspatches
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

OPatch succeeded.
  2. Copy gsm.ora, tnsnames.ora and the gsmwalletdirectory from the source host to the target host ($GSM_HOME/network/admin).
  3. Stop GSM2 on the source host.
  4. Modify the gsm.ora file with target host and target host wallet directory and modify the target host in tnsnames.ora file for the GSM2 entry.
  5. Modify the GSM configuration with the endpoint entry and verify using config gsm that it contais the correct target host details.
    GDSCTL> modify gsm -gsm gsm2 -endpoint (ADDRESS=(HOST=myhost.example.com)(PORT=1787)(PROTOCOL=tcp))
GSM modified
GDSCTL> 


GDSCTL> config gsm
Name      Region    ENDPOINT                                                    
----      ------    --------                                                    
gsm1      east      (address=(host=myhost.example.com)(port=1587)(protocol=tcp))                       
gsm2      west      (address=(host=myhost.example.com)(port=1787)(protocol=tcp))                       

GDSCTL>
  6. Start GSM2 on the target host.
    GDSCTL> start gsm -gsm gsm2;

GSM is started successfully

GDSCTL>
  7. Execute the modify gsm -gsm <gsm name> -pwd <GSMCATUSER password> command:
    GDSCTL> modify gsm -gsm gsm2 -pwd <GSMCATUSER secret_password>
GSM modified
GDSCTL>
  8. Verify the new GSM environment.
    GDSCTL> config database;
Name                Pool                Status    State       Region    Availability 
----                ----                ------    -----       ------    ------------ 
cloud               sdbpool             Ok        none        east      ONLINE       
clouddb             sdbpool             Ok        none        west      ONLINE       

GDSCTL>


GDSCTL> config

Regions
------------------------
east                          
west                          

GSMs
------------------------
gsm1                          
gsm2                          

GDS pools
------------------------
sdbpool                       

Databases
------------------------
cloud                         
clouddb                       

Services
------------------------
srv1                          

GDSCTL pending requests
------------------------
Command                       Object                        Status                        
-------                       ------                        ------                        

Global properties
------------------------
Name: orasubbu
Master GSM: gsm1
DDL sequence #: 0


GDSCTL


GDSCTL> status database;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: Y Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: Y Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL>


GDSCTL> databases;
Database: "cloud" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: east
   Service: "srv1" Globally started: Y Started: Y
            Scan: Y Enabled: Y Preferred: Y
   Registered instances:
     sdbpool%1
Database: "clouddb" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: west
   Service: "srv1" Globally started: Y Started: N
            Scan: Y Enabled: Y Preferred: N
   Registered instances:
     sdbpool%11

GDSCTL>


GDSCTL> status service;
Service "srv1.sdbpool.orasubbu" has 1 instance(s). Affinity: ANYWHERE
   Instance "sdbpool%1", name: "cloud", db: "cloud", region: "east", status: ready.

GDSCTL>


GDSCTL> config gsm
Name      Region    ENDPOINT                                                    
----      ------    --------                                                    
gsm1      east      (address=(host=myhost.example.com)(port=1587)(protocol=tcp))                       
gsm2      west      (address=(host=myhost.example.com)(port=1787)(protocol=tcp))                       

GDSCTL>

4.7 Password Management in a GDS Environment

Changing the GSMCATUSER Password

To change GSMCATUSER password:

  1. Run the following command in SQL*Plus while connected to the GDS catalog:
    ALTER USER gsmcatuser IDENTIFIED BY ****
  2. Then in GDSCTL run the following command:
    GDSCTL> modify catalog -oldpwd oldpassword -newpwd newpassword

The GSMUSR passwords are stored the GDS catalog in an encrypted form using the PKCS 1 encryption/decryption schema. You can encrypt GSMUSR passwords stored in the GDS catalog with a newly generated keys by executing the modify catalog command. For example: 

GDSCTL> modify catalog -newkeys

GSMCATUSER and GSMUSER accounts are shared by all global service managers in the Global Data Services framework and used for all management operations performed by global service managers, including automatic operations such as service failover. Human users should never connect to databases using these accounts.

In addition to the GSMCATUSER and GSMUSER accounts, the GSMADMIN_INTERNAL account is also used in a GDS configurations, both in the catalog and pool databases. This account's only purpose is to own the tables, packages, and other objects needed to support a GDS installation. It should never be unlocked, assigned a password, or used for interactive logins.

In a sharded invironment it is important to manage password rotation for the GSMUSER, GSMCATUSER, and GSMROOTUSER users. For a detailed instructions on performing this management task, navigate to Oracle Support and reference Doc ID 3052933.1.