If you encounter issues starting the shard director, try the following:

To start Scheduler you must be inside ORACLE_HOME on each shard server.

GDSCTL>start gsm -gsm shardDGdirector
GSM-45054: GSM error
GSM-40070: GSM is not able to establish connection to GDS catalog

GSM alert log, /u01/app/oracle/diag/gsm/shard1/sharddgdirector/trace/alert_gds.log
GSM-40112: OCI error. Code (-1). See GSMOCI trace for details.
GSM-40122: OCI Catalog Error. Code: 12514. Message: ORA-12514: TNS:listener does not 
currently know of service requested in connect descriptor
GSM-40112: OCI error. Code (-1). See GSMOCI trace for details.
2017-04-20T22:50:22.496362+05:30
Process 1 in GSM instance is down
GSM shutdown is successful
GSM shutdown is in progress
NOTE : if not message displayed in the GSM log then enable GSM trace level to 16 
while adding GSM itself.

1. Remove the newly created shard director (GSM) that failed to start.

   GDSCTL> remove gsm -gsm shardDGdirector
   

2. Add the shard director using trace level 16.

   GDSCTL> add gsm -gsm shardDGdirector -listener port_num -pwd gsmcatuser_password
    -catalog hostname:port_num:shard_catalog_name
    -region region1 -trace_level 16

3. If the shard catalog database is running on a non-default port (other than 1521), set the remote listener.

   SQL> alter system set local_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)
   (HOST=hostname)(PORT=port_num)))';

A failure in tablespace set creation may be due to DB_FILES parameter set too low.

DB_FILES parameter default setting is 200. This may be too low for distributed databases with a large number of shards and chunks. You may also require a larger number of data files in a Raft replication scenario

To calculate the number of database files created for distributed database objects on a given shard:

Distributed database files required = (Number of CREATE TABLESPACE SET SQL statements executed using SHARD DDL) * (Number of chunks present on the shard + 1)

DB_FILES must be set to at least the number of files used by the distributed database (above) PLUS non-distributed database files (system, sysaux, and so on) PLUS any extra needed by generic RDBMS code (5); therefore:

DB_FILES required in each shard = (Number of distributed database files required, as calculated above) + Number of default database files(6) + 5

To check free DB_FILES and parameter setting:

SQL> select count(*) from v$datafile;

  COUNT(*)
----------
XxxXX

SQL> show parameter db_files

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
db_files			     integer	 200

GDSCTL> deploy
GSM-45029: SQL error
ORA-29273: HTTP request failed
ORA-06512: at "SYS.DBMS_ISCHED", line 3715
ORA-06512: at "SYS.UTL_HTTP", line 1267
ORA-29276: transfer timeout
ORA-06512: at "SYS.UTL_HTTP", line 651
ORA-06512: at "SYS.UTL_HTTP", line 1257
ORA-06512: at "SYS.DBMS_ISCHED", line 3708
ORA-06512: at "SYS.DBMS_SCHEDULER", line 2609
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_POOLADMIN", line 14284
ORA-06512: at line 1

Solution : Check the $ORACLE_HOME/data/pendingjobs for the exact error. ORA-1017 is thrown if any issues on wallet.

1. On the problematic shard host stop the remote scheduler agent.

   schagent -stop

2. Rename wallet directory on the database home.

   mv $ORACLE_HOME/data/wallet $ORACLE_HOME/data/wallet.old

3. Start the remote scheduler agent and it will create new wallet directory.

   schagent -start 
   schagent -status 
   echo welcome | schagent -registerdatabase 10.10.10.10 8080

If you encounter issues with MOVE CHUNK, try the following:

Issue: Initialization parameter remote_dependencies_mode has a default value of timestamp; therefore, because prvtgwmut.plb is run and DBMS_GSM_UTILITY recompiled durning upgrade, GDSCTL MOVE CHUNK runs into ORA-04062 errors similar to the following.

GSM Errors:
server:ORA-03749: Chunk move cannot be performed at this time.
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_DBADMIN", line 5497
ORA-04062: timestamp of package "GSMADMIN_INTERNAL.DBMS_GSM_UTILITY" has been
changed
ORA-06512: at line 1
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_DBADMIN", line 5366
ORA-06512: at line 1 (ngsmoci_execute)

Workaround 1: Restart the source and target shards after upgrade.

Workaround 2: ALTER SYSTEM SET remote_dependencies_mode=signature on both source and target.