Troubleshooting

Internal Server Error in the Exadata Cloud discovery wizard

This error usually occurs in the step where you select the Monitoring Agent - OCI Credential Name, and the Home Region in the wizard.

Check the agent logs gcagent.log and look for entries in the format http://<service>.<region_identifier>.<realm_domain>.

The Exadata cloud discovery flow makes use of three OCI services, Database Service API, Identity and Access Management API and Search Service API. For example, https://identity.us-ashburn.oraclecloud.com, https://database.us-ashburn.oraclecloud.com, and https://query.us-ashburn.oraclecloud.com.

Verify that the OCI API service host names are reachable from agent host.

If they are not reachable, then verify the correctness of the URL. The OCI regions can belong to three different realms - commercial realm, Oracle Government Cloud realms, dedicated realms. The API URL formats are different for each realm. If incorrect URL is used, then contact Oracle Support to get the required patch.
If proxy is used, then see Set Up a Proxy Connection.
1. Ensure that the following properties are set in $EMSTATE/config/emd.properties on the agent:
```
oci_http_proxy_port=<proxy port>
oci_http_proxy_host=<proxy url>
```
  Optionally, set the properties oci_proxy_username and oci_proxy_password if authenticated proxy is used.
  
  If connection using proxy host fails, then try using the proxy host IP address instead to rule out DNS related issues.
2. Edit the file $EMDROOT/work/agentStateDir/sysman/config/s_jvm_options.opt and add the following line at end of the file and save the file:
```
-Djdk.http.auth.tunneling.disabledSchemes=""
```

Verify the OCI credentials by running the following OCI CLI command on the agent host:

oci db vm-cluster list --compartment-id <compartment id> --exadata-infrastructure-id <exadata infrastructure id>

Ensure that the profile information is present in ~/.oci/config file before running the OCI CLI command. For example:

[DEFAULT]
    user=<user ocid>
    fingerprint=<finger print>
    tenancy=<tenancy ocid>
    key_file=<pem key file path>
    region=<region code for example: us-ashburn-1>

Verify that the SSL certificate sent by OCI Service and the one received at the EM agent side are from trusted source - Oracle Corporation / Digicert. Run the following command on the agent host:

[oracle@<server>]$ openssl s_client -connect database.us-ashburn-1.oraclecloud.com:443
   CONNECTED(00000003)
   depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
   verify return:1
   depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
   verify return:1
   ---
   Certificate chain
   0 s:/C=US/ST=California/L=Redwood City/O=Oracle Corporation/CN=*.us-ashburn-1.oc-test.com
     i:/C=US/O=DigiCert Inc/CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
   1 s:/C=US/O=DigiCert Inc/CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
     i:/C=US/O=DigiCert Inc/OU

If the certificate is not from a trusted Certification Authority, for example, a self-signed certificate, and the proxy or firewall is replacing it with another certificate, then import the proxy or firewall certificate into the agent's JVM trust store (not keystore):

Copy the certificate crt file on the agent host.

Change to the right directory:

cd $EMDROOT/oracle_common/jdk/jre/lib/security/

Import the certificate:

keytool -importcert -keystore ./cacerts -file <certificate file> -alias <alias_name> -storepass <password>

Restart the agent:
```
emctl stop agent; emctl start agent
```

List of members of the cloud target does not show Database Instance

From cloud target home page, use the left navigation icon to view the navigation panel. Expand the list of available targets in the navigation panel, locate your database instance, and check if it belongs to a cluster database.

If the database belongs to a cluster database, then this is an expected behavior. It is not associated to the cloud target explicitly as a system member.

List of members of the cloud target does not show Cluster DB

All the hosts that are part of the cluster must be mentioned in the discovery properties file. To resolve the issue, add the missing host in the refresh procedure properties file and refresh the cloud target. See Refresh the Cloud Target After Discovery.

List of members of the cloud target does not show Storage Servers

Discovery is usually skipped in one of the following scenarios:

Credentials are not supplied in the discovery properties file. See Discover the Cloud Target Using emcli.
IP's of the storage servers are typically listed in the file /etc/oracle/cell/network-config/cellip.ora which is located in one of the hosts mentioned in the discovery properties file. Discovery will be skipped if the file is not present on any of the hosts.
Credentials are incorrect. See Create Storage Server Credentials.
exacli is not installed on any of the hosts.

To resolve the issue, try one of the following:

Ensure that proper named credentials are created and provided in the property file.
Ensure that exacli is installed on each of the hosts listed in the property file that was used for discovering the cloud target.

Cell Admin command fails for RestAPI monitored cells

This error occurs when the agent is running with the user that is different from the user who installed the agent. For RestAPI monitoring mechanism, the HOST PDP settings are mandatory.

To resolve the issue, set the privilege delegation for the user that is running the agent. To set the Host PDP settings, click Setup Menu, click Security, and select Privilege Delegation.

Discovery of Exadata Storage Server failed when $HOME/.exacli/cookiejar is present

The discovery of the Exadata Storage Server fails when the cookie-jar is present at $HOME/.exacli/cookiejar. Note that cookie-jar is not supported for Enterprise Manager monitoring of Exadata Storage Server.

To resolve the issue, move cookie-jar from $HOME/.exacli/cookiejar and attempt the discovery of the Exadata Storage Server.