Troubleshoot the Connectivity Agent

Topics:

Troubleshoot Agent Unavailable Issues

Connectivity agent status can be determined by selecting Observability > Agents to access the Monitor agents page. If the agent status is displayed as unavailable, follow this troubleshooting workflow.

Number	Step	Details
1	Check the agent process is running	Check whether the agent is running. Run one of these commands from the host where the agent is installed. If you don't see `connectivityagent.jar` as a listed process, the agent is not running. For example: `19054 connectivityagent.jar` Linux: `ps –ef \| grep connectivityagent.jar` Windows: `jps -l` If the agent process is not running, start it: Linux: Start the connectivity agent as a background process: `nohup java -jar connectivityagent.jar &` Windows: `java -jar connectivityagent.jar` Note: Starting the agent as a Windows service is not supported. If you cannot start the agent, continue to check network connectivity. If the agent process is running, restart it: Follow these steps to restart the agent.
2	Check network connectivity	Check that the agent host can connect to the Oracle Integration instance: On the host where the agent is installed, run this command, replacing with your own path: `curl https://myoic-tenancy-ia.integration.ocp.oraclecloud.com/ic/home` If the response is 302 Found, the agent is able to resolve the Oracle Integration instance host. You have network connectivity. If the response is not 302 Found, there may be network issues. Check the `/etc/hosts` file: Check for the correct IP address and host name for your Oracle Integration instance. Troubleshoot with your network administrator to check that connectivity to the instance is not blocked by a firewall or proxy. Here's agent troubleshooting information.
3	Download the Configuration File	Download the agent configuration file again and restart the connectivity agent after replacing the previous agent configuration file.
4	Log a Service Request	If you've gone through the troubleshooting steps, can't find anything in the diagnostic log, and the agent status is still unavailable, create a service request with Oracle Support. Include the diagnostic log in the service request.

Unable to Start Connectivity Agent Because Agent Status Cannot Be Obtained

Connectivity agent startup can fail with the following error:

java -jar connectivityagent.jar &

[2024-12-09T19:30:16.392Z] [INFO] [ThreadID: 15] [Logger: oracle.wsm.agent.WSMAgent] [SRC Class: oracle.wsm.agent.WSMAgent; Method: processRequest] pool-1-thread-1Entering processRequest() -  12/09/2024 19:30:16:392
[2024-12-09T19:30:16.396Z] [INFO] [ThreadID: 15] [Logger: oracle.wsm.agent.WSMAgent] [SRC Class: oracle.wsm.agent.WSMAgent; Method: processRequest] pool-1-thread-1Exiting processRequest() -  12/09/2024 19:30:16:396
[2024-12-09T19:30:16.473Z] [INFO] [ThreadID: 15] [Logger: oracle.cloud.cpi.agent.util.CpiUtils] [SRC Class: oracle.cloud.cpi.agent.util.CpiUtils; Method: createAuthToken] OAuth token url https://idcs-xxxxxx.identity.oraclecloud.com:443/oauth2/v1/token
[2024-12-09T19:30:46.505Z] [SEVERE] [ThreadID: 15] [Logger: oracle.cloud.cpi.agent.util.CpiUtils] [SRC Class: oracle.cloud.cpi.agent.util.CpiUtils; Method: createAuthToken] IDCS Token Generation: Call to create token failed due to java.net.SocketTimeoutException: Connect timed out
[2024-12-09T19:30:46.507Z] [INFO] [ThreadID: 15] [Logger: oracle.wsm.agent.WSMAgent] [SRC Class: oracle.wsm.agent.WSMAgent; Method: processRequest] pool-1-thread-1Entering processRequest() -  12/09/2024 19:30:46:507
[2024-12-09T19:30:46.507Z] [INFO] [ThreadID: 15] [Logger: oracle.wsm.agent.WSMAgent] [SRC Class: oracle.wsm.agent.WSMAgent; Method: processRequest] pool-1-thread-1Exiting processRequest() -  12/09/2024 19:30:46:507
[2024-12-09T19:30:46.550Z] [SEVERE] [ThreadID: 15] [Logger: oracle.cloud.cpi.agent.install.AgentInstaller] [SRC Class: oracle.cloud.cpi.agent.install.AgentInstaller; Method: checkVersion] Agent Installation Failed !!! Incorrect OIC Username / Password provided.
[2024-12-09T19:30:46.551Z] [SEVERE] [ThreadID: 15] [Logger: oracle.cloud.cpi.agent.ProxyCpiAgentLoader] [SRC Class: oracle.cloud.cpi.agent.ProxyCpiAgentLoader; Method: configureAgent] Agent Installation Failed !!! java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethod

This error occurs because the connectivity agent cannot query the agent status from the Oracle Integration server.

Allowlist your identity domain or Oracle Identity Cloud Service, design time IP address, and runtime IP address.
Check connectivity to Oracle Integration 3.
(Optional) Download the configuration file again and restart the agent if no issue is found with Oracle Integration 3 connectivity.

Connectivity Agent Startup Fails With a "java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status" Error

Connectivity agent startup can fail with the following error:


[2021-06-19T10:06:06.567Z] [SEVERE] [ThreadID: 1] [Logger: oracle.cloud.cpi.agent.CpiBootstrapLoader] 
[SRC Class: oracle.cloud.cpi.agent.CpiBootstrapLoader; Method: main] Error encountered !!! java.lang.RuntimeException: 
 java.lang.reflect.InvocationTargetException
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:452)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiBootstrapLoader.startAgent(CpiBootstrapLoader.java:192)
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:436)
...
...
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiAgentLoader.startAgent(CpiAgentLoader.java:27)
...
...
Caused by: java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status :
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.initialize(ProxyCpiAgentLoader.java:151)
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.runAgent(ProxyCpiAgentLoader.java:53)
...
...

Ensure that connectivity from your agent host to Oracle Integration 3 is working.

Allowlist your identity domain or Oracle Identity Cloud Service, design time IP address, and runtime IP address.
Check connectivity to Oracle Integration 3.
(Optional) Download the configuration file again and restart the agent if no issue is found with Oracle Integration 3 connectivity.

Connectivity Agent Installation Error if /etc/hosts File is Not Configured

During connectivity agent installation, if the agent installer cannot find details about the host name/virtual machine (VM) on which the agent is being installed, you may receive the following error:

Caused by: java.net.UnknownHostException: : Unknown
name or service

Resolve this error by adding an entry for the IP address and host name in the /etc/hosts file of the host name/VM.

Restrictions on Using the Stage File Action with the Connectivity Agent

Be aware of several restrictions when using some operations of the stage file action with the connectivity agent. See Restrictions on Using Stage File Action Operations with the File/Attachment Features of the Connectivity Agent.

Java Memory Errors

The agent Java Virtual Machine can stop running when allocating memory with the following seemingly out-of-memory error:

There is insufficient memory for the Java Runtime Environment to continue.
-----------------------------
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007f6847afd000, 12288, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing
reserved memory.
# An error report file with more information is saved as:

However, this issue is unrelated to the out-of-memory error that is sometimes observed when the Java heap is not large enough.

This error occurs when Java requests more memory from the operating system, which does not have any:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map

This may be related to resources on the physical machine/virtual machine/server on which the agent is running. As an example, when the agent is running on the same virtual machine as the database, and the database may be consuming most of the resources. Oracle recommends that you set up the agent on a separate compute.

Add or Change the Non-Proxy Host Configuration After Installation in the CpiAgent.properties File

If you need to add, change, or bypass the non-proxy host configuration after agent installation, do not edit the InstallerConfing.cfg file. Proxy host changes made to that file after agent installation do not take effect. Instead, update the host with the proxy_nonProxyHosts parameter in the Agent_Installation_Location/agenthome/agent/config/CpiAgent.properties file for your changes to take effect. After editing this file, restart the agent.

Class Loading Conflict When Sharing the JDK Instance with Another Product

When using the connectivity agent, the following error occurs because your JDK instance is shared with another product for which JAR files have been added in the JDK's endorsed directory. This results in a class loading conflict with the agent:

ClassCastException: com.sun.xml.messaging.saaj.soap.ver1_1.Message1_1Impl
cannot be cast to oracle.j2ee.ws.saaj.soap.MessageImpl

Ensure that your connectivity agent is running with a JDK installation that is not modified because of use with other products.

Connectivity Agent Log File Location

The connectivity agent agent-diagnostic0.log file is available under agenthome/logs.

Agent Behavior in a Decommissioned Instance or HTTP 404/401 Error Response Codes

The following code in the logs indicates that agent runtime message processing has halted. This occurs if an HTTP 404/401 error code is continuously received by the agent for 24 hours. The decommission of an Oracle Integration instance also triggers this behavior. When the conditions leading to this error have been resolved, the agent must be restarted manually. See Restart the Connectivity Agent.

[2018-10-13T04:30:13.501Z] [SEVERE] [ThreadID: 18] [Logger:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer] [SRC Class:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer; Method: run] Terminate
flag activated. Signalling termination of agent runtime poller thread with Id

CPI_RUNTIME_REQ_RES_CHANNEL_worker_0.

Verify Endpoint Accessibility when Agent is Installed with a Proxy Host

When the agent is installed with a proxy host, carefully check that the endpoint to access through the agent is reachable through the proxy host. If it is not reachable through the proxy host, you must configure the on-premises endpoint host in the proxy_NON_PROXY_HOSTS parameter of the Agent_Installation_Location/agenthome/agent/config /CpiAgent.properties file.

Unlock the Agent Group

When an agent group is in edit mode and the browser crashes, the agent group becomes locked, which prevents it from being edited. This results in the following error:

ICS-10507: The agent group cannot be updated because it is locked.

To unlock the agent group:

Log in again as the same user who was editing the agent group when the browser crashed, then log out. This action unlocks the agent group.

or

Wait 30 minutes for the lock to expire after the timeout starts.

Failure to Send a Response Due to a java.net.SocketException: Connection Reset Error

For connectivity agent installations running on an Oracle Integration Classic VM and connecting to Oracle Integration (running on Oracle Cloud Infrastructure), the design-time and runtime operation involving the connectivity agent sometimes fails with a java.net.SocketException: Connection reset error.

This can occur because of a Maximum Transmission Unit (MTU) mismatch.

Here is the complete error:

[2019-01-03T16:35:12.670Z] [SEVERE] [ThreadID: 50] [Logger:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter] [SRC Class:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter; Method:
sendOneWayPacket] Exception while sending response back to
ICSjava.net.SocketException: Connection reset
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException:
Connection reset
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConne
 
ctionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)

To make this connection work, set the MTU of the Oracle Integration Classic VM (where the agent is installed) to 1500 from the current value of 8900.

Perform the following steps:

Run ifconfig -a as the root user (sudo) and note down the network interface.
Run the following command as the root user (sudo) for the network interface (assuming the network interface is eth0):
```
ifconfig eth0 mtu 1500
```

Anytime the agent VM is restarted (note that it is not an agent restart, but the host where the agent is installed), the changes must be done for the network interface before restarting the agent.

Agent Installation on Linux Fails When Using an Installer Copied Using winscp

During connectivity agent installation in Linux environments, the installation sometimes fails with the following error:

On premise agent is throwing the following error: java.lang.RuntimeException:

Agent Startup Failed - java.lang.IllegalArgumentException: URI is not
absolute

Installation failure occurs for the following reason:

The agent is to be installed in a Linux environment.
The agent installer ZIP file is downloaded in a Windows environment and transferred to a Linux environment for installation using a Windows tool called winscp.
Even when the binary option is enabled in the winscp tool, the installer ZIP used to run and install the agent fails with the above error.

As a workaround, perform the following steps:

Download the agent installer to a Linux environment directly and do not transfer it from a Windows environment.
If the Oracle Integration user interface is accessible from a Linux environment, use the download install option provided on the Agents page.

If the Oracle Integration user interface instance is not accessible, use the following REST command to download the installer to a Linux environment:

curl -k -v -X GET -u OIC_user:OIC_password 
-H 'Content-Type:application/json'
'https://OIC_URL:443/icsapis/v1/agent/binaries/connectivity' 
-o download_location/oic_connectivity_agent.zip

Integration Activation Error Due to Change in JDK Location

If integration activation fails with the following error, this is likely the result of the agent installation using a JDK whose location has been changed (for example, removed). This can occur if the agent was installed and running with a version of the JDK whose location was removed and a newer version was installed in a different location. If the JDK installed with the agent is removed, ensure that you restart the agent with the newer version (and location) of the JDK.

Caused by: java.lang.Error: Circular loading of installed providers 
detected at 
java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.jav 
a:161)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:324)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
at
oracle.cloud.cpi.agent.store.StoreUtils.extractZipBundle(StoreUtils.java:49)
at
oracle.cloud.cpi.agent.store.FileAgentStore.addActivation(FileAgentStore.java:
40)
at
oracle.cloud.cpi.agent.ActivationCpiCommand.doRequest(ActivationCpiCommand.jav 
a:80)

Troubleshoot Network Connectivity Issues

Ensure that network connectivity is working correctly when persistent connectivity failures are encountered with an Oracle Integration instance.

Note:

This does not apply to any intermittent failures because the connectivity agent is resilient to temporary conditions and recovers when the situation is resolved.

Run the following command:
```
nslookup hostname
```

Run the following command for a period of five minutes to also capture any transient failures:

while true;
do
curl https://Oracle_Integration_hostname/ic/home >> file.txt 2>&1;
sleep 1;
echo "trying again";
done;

HTTP 401 Unauthorized Error Occurs During Connectivity Agent Installation

If connectivity agent installation or restart fails with an HTTP 401 Unauthorized error, download the agent configuration file from Oracle Integration 3 and restart the connectivity agent.