1. Create a logging.properties file. The following example configures the JDK logger to emit messages to both the console and to a file. Output to the console and file is configured to use the FINEST log level. For the file handler pattern, the specified path must already exist. Also, ensure that the specified directory can be accessed and has write permissions.

   handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler
   
   .level=INFO
   Coherence.level=FINEST
   
   java.util.logging.FileHandler.pattern=/tmp/coherence%u.log
   java.util.logging.FileHandler.limit=50000
   java.util.logging.FileHandler.count=1
   java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
   
   java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
   

   Note:

   • In the above example, Coherence is used as the logger object name and is the default name that is used by the Coherence logging framework. A different name can be used by specifying the name within the <logger-name> element in the operational override file or by specifying the name as the value of the coherence.log.logger system property.

   • Set the JDK logging level to FINEST and allow the Coherence logging settings to determine which log messages to construct for JDK logging.

2. Configure Coherence to use JDK logging by specifying jdk as the value of the <destination> element in an operational override file. For example:

   ...
   <logging-config>
      <destination system-property="coherence.log">jdk</destination>
      ...
   </logging-config>
   ...
   

3. Make sure the logging.properties file is specified using the java.util.logging.config.file system property. For example:

   -Djava.util.logging.config.file=myfile
   

Table 5-2 provides a mapping of how Log4j2 log levels are mapped to Coherence log levels.

1. specify slf4j as the value of the <destination> element in an operational override file. For example:

   ...
   <logging-config>
      <destination system-property="coherence.log">slf4j</destination>
      ...
   </logging-config>
   ...
   

2. Make sure both the slf4j-api.jar file and the appropriate logging framework SLF4J binding JAR is located on the classpath.

To configure tracing, edit the operational override tangosol-coherence-override.xml file and add a <tracing-config> element with a child <sampling-ratio> element.

For example:

...
<tracing-config>
  <sampling-ratio>0</sampling-ratio> <!-- user-initiated tracing -->
</tracing-config>
...

Tracing operates in three modes:

• -1 - This value disables tracing.
• 0 - This value enables user-initiated tracing. This means that Coherence will not initiate tracing on its own and the application should start an outer tracing span, from which Coherence will collect the inner tracing spans. If the outer tracing span is not started, the tracing activity will not be performed.
• 0.01-1.0 - This range indicates the tracing span being collected. For example, a value of 1.0 will result in all spans being collected, while a value of 0.1 will result in roughly 1 out of every 10 spans being collected.

The coherence.tracing.ratio system property can be used to specify the tracing sampling ratio instead of using the operational override file. For example:

-Dcoherence.tracing.ratio=0

When the sampling ratio is set to zero, the application will be required to start a tracing span prior to invoking a Coherence operation.

For example:

Tracer tracer = GlobalOpenTelemetry.getTracer(“your-tracer");

Span span = tracer.spanBuilder("test").startSpan();

NamedCache cache = CacheFactory.getCache("some-cache")

try (Scope scope = span.makeCurrent())
    {
    cache.put("a", "b");
    cache.get("a");
    }
finally
    {
    span.end();
    }

Log messages provide information that is used to monitor and troubleshoot Coherence. See Log Message Glossary in Administering Oracle Coherence. The glossary provides additional details as well as specific actions that can be taken when a message is encountered.

Configuring logging beyond the default out-of-box configuration is very important when developing and debugging an application. Specifically, use the highest log level (level 9 or ALL when using JDK or Log4j2 logging) to ensure that all log messages are emitted. Also, consider using either JDK or Log4j2 logging. Both of these frameworks support the use of rolling files and console output simultaneously. Lastly, consider placing all log files in a common directory. A common directory makes it easier to review the log files and package them for the Coherence support team. See Configuring Logging.

Coherence management is implemented using Java Management Extensions (JMX). Many MBeans are provided that detail the health and stability of Coherence. The MBeans provide valuable insight and should always be used when moving an application from a development environment to a fully distributed environment. MBeans are accessible using JConsole and VisualVM or any management tool that supports JMX. In addition, Coherence includes reports that gather information from the MBeans over time and provide a historical context that is not possible simply by monitoring the MBeans. The reports are most often used to identify trends that are valuable for troubleshooting. Management and reporting are not enabled by default and must be enabled. See Configuring JMX Management and Enabling Oracle Coherence Reporting on a Cluster Member in Managing Oracle Coherence.

Most JVMs include options that facilitate debugging and troubleshooting. These options should be used to get as much information as possible. Consult your JVM vendor's documentation for their available options. The JVM options discussed in this section are Java HotSpot specific. See the Java HotSpot VM Options Web page.

The following JVM options (standard and non standard) can help when debugging and troubleshooting applications:

• -verbose:gc or -Xloggc:file – These options are used to enable additional logs for each garbage collection event. In a distributed system, a GC pause on a single JVM can affect the performance of many JVMs, so it is essential to monitor garbage collection very closely. The -Xloggc option is similar to verbose GC but includes timestamps.

• -Xprof and -Xrunhprof – These options are used to view JVM profile data and are not intended for production systems.

• -XX:-PrintGC, -XX:-PrintGCDetails, and -XX:-PrintGCTimeStamps – These options are also used print messages at garbage collection.

• -XX:-HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=./java_pid<pid>.hprof – These options are used to initiate a heap dump when a java.lang.OutOfMemoryError is thrown.

• -XX:ErrorFile=./hs_err_pid<pid>.log – This option saves error data to a file.

Distributed tracing allows developers to profile and monitor applications. This is particularly useful in a clustered environment such as Coherence. See Distributed Tracing.

Thread dumps are used to see detailed thread information, such as thread state, for each thread in the JVM. A thread dump also includes information on each deadlocked thread (if applicable). Thread dumps are useful because of Coherence's multi-threaded and distributed architecture. Thread dumps are often used to troubleshoot an application that is operating slowly or is deadlocked. Make sure to always collect several dumps over a period of time since a thread dump is only snapshot in time. Always include a set of thread dumps when submitting a support issue.

Coherence provides a native logClusterState JMX operation that is located on the ClusterMBean MBean and a native logNodeState JMX operation that is located on the ClusterNodeMBean MBean. These operations initiate a thread dump (including outstanding polls) on multiple cluster members or on a single cluster member, respectively. See ClusterMBean and ClusterNodeMBean in Managing Oracle Coherence.

To perform a thread dump locally on Unix or Linux operating systems, press Ctrl+\ at the application console. To perform a thread dump on Windows, press Ctrl+Break (or Pause). Both methods include a heap summary with the thread dump.

Most IDEs provide a thread dump feature that can be used to capture thread dumps while working in the IDE. In addition, Unix and Linux operating systems can use the kill -3 pid to cause a remote thread dump in the IDE. On Windows, use a third party tool (such as SendSignal) to send a ctrl+break signal to a remote Java process and cause a dump in the IDE.

Profiling tools, such as Oracle's VisualVM (visualvm) and JConsole (jconsole) are able to perform thread dumps. These tools are very useful because they provide a single tool for troubleshooting and debugging and display many different types of information in addition to just thread details.

Lastly, the jstack tool can be used to capture a thread dump for any process. For example, use jps to find a Java process ID and then execute the following from the command line:

jstack <pid>

The jstack tool is unsupported and may or may not be available in future versions of the JDK.

The following WLST script can be used to trigger cluster wide thread dumps when running Coherence in a WebLogic domain:

# connect to domain runtime mbean server
connect(adminUser, adminPasswd, "t3://%s:%s" % (adminHost, adminPort))
domainRuntime()

# obtain coherece cluster mbean
cohClusterON=ObjectName("Coherence:type=Cluster,cluster=%s" % cohClusterName)
cohClusterMBean=list(mbs.queryMBeans(cohClusterON, None))

# obtain all mbean of coherence nodes associated with this cluster
cohClusterNodes=list(mbs.queryMBeans(ObjectName("Coherence:type=Node,cluster=%s,*" % cohClusterName), None))

# take a thread dump on each coherence member
types=["java.lang.String"]
for node in cohClusterNodes:
  roleName=mbs.getAttribute(node.getObjectName(), "RoleName")
  mbs.invoke(cohClusterMBean[0].getObjectName(), 'logClusterState', [roleName], types)

Heap dumps are used to see detailed information for all the objects in a JVM heap. The information includes how many instances of an object are loaded and how much memory is allocated to the objects. Heap information is typically used to find parts of an application that may potentially be wasting resources and causing poor performance. In a fully distributed Coherence environment, heap dumps can be tricky because application processing is occurring across the cluster and problematic objects may not necessarily be local to a JVM. Make sure to always collect several dumps over a period of time since a heap dump is only a snapshot in time. Always include heap dumps when submitting a support issue.

The easiest way to capture a heap dump is to use a profiling tool. Oracle's VisualVM (visualvm) and JConsole (jconsole) provide heap dump features. In addition, most IDEs provide a heap dump feature that can be used to capture heap dumps while working in the IDE.

As an alternative, the jmap tool can be used to capture heap dumps, and the jhat tool can be used to view heap dumps. For example, use jps to find a Java process ID and then execute the following from the command line:

jmap -dump:format=b,file=/coherence.bin pid

To view the heap dump in a browser, execute the following from the command line and then browse to the returned address. The file can also be loaded into VisualVM for viewing.

jhat /coherence.bin

The jmap and jhat tools are unsupported and may or may not be available in future versions of the JDK.

Always monitor a cluster member's operating system when troubleshooting and debugging Coherence-based applications. Poorly tuned operating systems can affect the overall performance of the cluster and may have adverse effects on an application. See Operating System Tuning in Administering Oracle Coherence.

In particular, the following areas are important to monitor:

• CPU – Is the processor running at 100% for extended periods of time?

• Memory/Swapping – Is the available RAM memory being exhausted and causing swap space to be used?

• Network – Is buffer size, the datagram size, and the Maximum Transmission Unit (MTU) size affecting performance and success rates?

To monitor the overall health of the operating system, use tools such as vmstat and top for Unix/Linux; for Windows, use perfmon and tools available from Windows Sysinternals (for example procexp and procmon). See Performing a Network Performance Test in Administering Oracle Coherence.