Getting Started with Management Pack for Oracle Coherence

19 Getting Started with Management Pack for Oracle Coherence

This chapter describes the procedure to discover and monitor a Coherence cluster using Oracle Enterprise Manager Cloud Control 13c. The following sections are covered in this chapter:

About Coherence Management

Oracle Coherence is an in-memory data-grid and distributed caching solution. It is composed of many individual nodes or java processes which work together to provide highly reliable and high speed virtual caching.

Enterprise Manager provides deep visibility into performance of all the artifacts such as caches, nodes, and services. Nodes and caches can be proactively monitored by the Incident Management feature. You can create a monitoring template by pre-populating the monitoring template with metrics for a Coherence target. You can export and import monitoring templates to share monitoring settings between different Enterprise Manager deployments.

Metric Extensions are the next generation of User-Defined Metrics, which enable you to extend Enterprise Manager to monitor conditions specific to the enterprise's environment by creating new metrics for any target type. By including metric extensions in export or imported monitoring templates, multiple metric extensions can be easily shared at the same time between Enterprise Manager deployments.

You can correlate cluster nodes with the underlying hosts to determine CPU and memory utilization on those hosts in order to make better decisions for scaling your clusters. You can see the association between the caches, nodes, hosts, and Oracle WebLogic targets.

Highly customizable performance views for monitoring performance charts and trends are available. You can overlay metrics for multiple nodes or caches in the same or different cluster for detail analysis to provide detailed visibility at the desired level. The drill down views allows you to determine the root cause of performance problems or simply identify performance trends in the Coherence Cluster.

Enterprise Manager provides a centralized cache data management feature that allows you to perform various cache operations such as add/remove index, view cache data, view query explain plan, and so on.

Enterprise Manager monitors the changing configuration of the nodes over a period of time. The Topology Viewer provides a high level topology of the entire cluster and shows the relation between caches, nodes and hosts.

All of the Coherence Management features are integrated with JVM Diagnostics and provide real-time visibility into the node JVMs. You can drill down to a Coherence node's JVM from within the context of a cache and a cluster to identify the method or thread that is causing a delay. The JVM Diagnostics feature is part of the WLS Management Pack EE and Management Pack for NonOracle Middleware.

Enterprise Manager provides a complete provisioning solution. You can maintain an Oracle Coherence setup image or gold image in the Software Library and deploy it throughout the infrastructure to create completely new clusters or add nodes an existing cluster. You can use the same deployment procedure to updates nodes as well.

New Features for Oracle Coherence

This section lists the new features in Oracle Enterprise Manager Cloud Control 13c. The new features are:

Target Navigation: The target navigation menu is hidden by default. Click the Target Navigation icon on the left hand side of the page to display the Target Navigation tree. The navigation tree makes it easier to navigate to any node or cache from any page. Nodes are grouped based on the hosts on which they are running, and caches are grouped based on services. See Navigation Tree.
Heat Map: The Cluster Home page now has a Heat Map tab which provides a graphical representation of all the targets in the cluster.
Log Viewer: You can now scan the log files and view the log file data.
Remove Down Members: You can now delete any members of the cluster that have a Down status.
Managed Coherence Clusters: The discovery and monitoring of managed Coherence clusters has now been integrated with Oracle Fusion Middleware/Weblogic Domain discovery and monitoring.
Custom Topology Viewer: The custom topology viewer provides a customized view of all the targets in a Coherence cluster.
Federated Caching: You can synchronize cache instances across clusters by replicating updates from a cache in a source cluster to a cache in a remote cluster.
Discovery of Dynamic Management Node: The coherence 12.2.1.x and above can be configured to start in dynamic management mode. Dynamic management mode automatically selects the senior cluster member as the management node (JMX cluster member). If the central management node is not operational, then the next most senior coherence node is automatically selected as the management node.
Federated Operations: A new Federated Caching Operations region is added to Service Home Page. This allows the user to submit operation request. Based on operation selection, federation operation will be executed with reference to that remote participant.
Target Discovery: You can now disable cache target discovery during new discovery and refresh.

Configuring a Coherence Cluster

Note:

This section covers the configuration procedure for a standalone Coherence cluster.

For details on configuring a managed Coherence cluster, refer to the WebLogic documentation.

Oracle Coherence standalone deployments can be monitored using Enterprise Manager by configuring the Coherence nodes with a set of Coherence and JMX system properties (start arguments). In addition, one of the nodes will have to be configured as a central JMX management node. This JMX management node must expose all Coherence MBeans and attributes. See Creating and Starting a JMX Management Node for details. In addition to configuring the JMX management node, the Management Agent must also be installed and configured on the same host as JMX management node. This is required to discover and monitor the Coherence cluster in Enterprise Manager.

Figure 19-1 shows the configuration for monitoring standalone Coherence clusters using Enterprise Manager.

Figure 19-1 Coherence Cluster Configuration (Standalone Coherence Cluster)

As shown in the figure, Coherence Management (JMX) node's MBean server will expose MBeans for entire Coherence cluster. Enterprise Manager will connect to this management node to discover and monitor Coherence cluster.

Creating and Starting a JMX Management Node

The Management Agent uses the JMX management node (centralized MBean server) to discover and monitor the entire Coherence cluster, including the nodes and caches. As a best practice, it is recommended that the Management Agent be present on the same host as the JMX management node that is used to discover and monitor the Coherence cluster. The Management Agent must be setup on all the machines on which the Coherence nodes are running to monitor and provision the cluster. For more information on using JMX to manage Oracle Coherence, see Using JMX to Manage Coherence in the Oracle Coherence Management guide. To configure the JMX management node, you must:

Specify Additional System Properties
Include Additional Class Path
Use the Enterprise Manager Custom Start Class

Specifying Additional System Properties

Note:

Oracle recommends that the management node is configured as a storage disabled node to ensure minimal performance impact on any Coherence caches.

The following start arguments must be added to one of the Coherence nodes to configure it as the JMX central management node.

-Dtangosol.coherence.management.extendedmbeanname=true (allows any restarted node to be automatically detected by Enterprise Manager. This parameter is available in Coherence 3.7.1.9 and later versions)
- If set to true, the status of the node is automatically refreshed when a node is restarted.
- If this property is not set, you must use the Refresh Cluster option to update the status of a node when it is restarted.
- If you start a node after setting this property to true, all nodes in the cluster must be started after the extendedmbeanname property is set to true.
-Dtangosol.coherence.management=all (enables monitoring for all nodes)
-Dcom.sun.management.jmxremote.port=<port number> (required for remote connection for coherence 12.2.1.x or older versions)
-Dtangosol.coherence.distributed.localstorage=false (disables caching and ensures that the node is a dedicated monitoring node)
-Doracle.coherence.home=<coherence home>
-Dtangosol.coherence.member=<member name> (required for target name)
-Doracle.coherence.machine=<fully qualified hostname> (must match the name of the host discovered in Enterprise Manager)

Note:

If you are using JMX credentials, you must set the following additional start arguments.

-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.authenticate=true

If no JMX credentials are used, you must set these arguments to false.

Including the Additional Class Path

You must include the path to both Enterprise Manager custom jar files, coherenceEMIntg.jar and bulkoperationsmbean.jar.

For coherence cluster versions older than 12.2.1, the jar files are available in the

<OEM_Agent_Home>/<PLUGIN_HOME>/<MIDDLEWARE_MONITORING_PLUGIN_DIR>/archives/coherence

directory.

Coherence cluster with version 12.2.1 and above, must use the coherenceEMIntg.jar file available in the

<OEM_Agent_Home>/<PLUGIN_HOME>/<MIDDLEWARE_MONITORING_PLUGIN_DIR>/archives/coherence\12.2.1

directory.

Note:

The location of the .jar files may change based on the plugin version.

Using the Custom Start Class

In addition to configuring the system properties and the class path when starting Coherence management node, it is also required that you use the Enterprise Manager EMIntegrationServer class as the start class. This class allows you to register the custom MBeans required for the Cache Data Management feature of Management Pack for Oracle Coherence.

Example Start Script for the Coherence Management Node

An example start script for the management node is given below:

#
!/bin/sh

CP=$CP:<EM CC_Agent_Home>/plugins/oracle.sysman.emas.agent.plugin_12.1.0.6.0/archives/coherence/coherenceEMIntg.jar:
<EM CC_Agent_Home>/plugins/oracle.sysman.emas.agent.plugin_12.1.0.6.0/archives/coherence/bulkoperationsmbean.jar
COH_OPTS="$COH_OPTS -cp $CP"
$JAVA_HOME/bin/java $COH_OPTS
-Dtangosol.coherence.management.extendedmbeanname=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.ssl=false 
-Dtangosol.coherence.management=all 
-Dtangosol.coherence.member=<unique member name> 
-Doracle.coherence.machine=<hostname_as_discovered_in_EM> 
-Dcom.sun.management.jmxremote.port=<OpenTCP_Port>
-Doracle.coherence.home=$COHERENCE_HOME 
-Dtangosol.coherence.distributed.localstorage=false 
-Dtangosol.coherence.management.refresh.expiry=1m 
-server 
-Xms2048m -Xmx2048m 
oracle.sysman.integration.coherence.EMIntegrationServer

Configuring All Other Nodes

In addition to configuring the Coherence JMX management node, you must configure all other Coherence cluster nodes with additional Coherence specific system properties (start arguments) used by Enterprise Manager.

Additional System Properties for All Other Coherence Nodes

The following system properties must be added to all other Coherence nodes.

-Dtangosol.coherence.management.extendedmbeanname=true
-Dtangosol.coherence.management.remote=true –Dtangosol.coherence.member=<unique member name> -Doracle.coherence.home=<coherence home>
-Doracle.coherence.machine=<machine name> should be the same as the name of the host discovered in Enterprise Manager.

Note:

If you are using JMX credentials, you must set the following additional start arguments.

-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.authenticate=true

If no JMX credentials are used, you must set these arguments to false.

Example Start Script for All Other Coherence Nodes

An example start script for all other Coherence nodes is given below:

#!/bin/sh
 
COH_OPTS="$COH_OPTS -cp $CP"
$JAVA_HOME/bin/java $COH_OPTS
-Dtangosol.coherence.management.extendedmbeanname=true
-Dtangosol.coherence.management.remote=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Doracle.coherence.home=<coherence home>
-Dtangosol.coherence.member=<unique member name> 
-Doracle.coherence.machine=<hostname> 
-Dcom.tangosol.net.DefaultCacheServer

Testing the Configuration

To test the Coherence cluster configuration for use in Enterprise Manager, you must verify that the central management (JMX) node has information regarding the managed objects of all other Coherence cluster nodes, caches, services, and so on. Additionally, you must verify that the central management node is accessible remotely, either through <hostname>:<port> OR the JMX Service URL. If JMX credentials are used, they should also be specified.

Verifying Remote Access for the MBean Objects Using JConsole

JConsole is a Java tool available through JDK. You can use this to verify remote access to the MBean objects of entire Coherence cluster nodes, caches, services, and so on.

Figure 19-2 JConsole

To verify remote access, open JConsole and select "New Connection". In New Connection page, select Remote Process and provide connection details where <hostname> is the name of the machine where central management node is running, <port> is what you have specified in the -Dcom.sun.management.jmxremote.port parameter while starting the management node. If successful, you will see the MBean object tree.

Figure 19-3 MBean Object Tree

If you see MBeans for all Coherence nodes in the System MBean Browser or JConsole, you can now discover and monitor the Coherence cluster and its associated elements in Enterprise Manager.

Discovering Coherence Targets

This section covers the following:

Discovering a Standalone Coherence Cluster

Enterprise Manager monitors the entire Coherence cluster and its artifacts. The key targets that can be monitored are Oracle Coherence Cluster, Oracle Coherence Node, and Oracle Coherence Cache. The Oracle Coherence Cluster target provides a high level view of the health of the entire cluster. The Oracle Coherence Node and Oracle Coherence Cache are child targets of the Oracle Coherence Cluster. In addition to monitoring the above target types, additional Coherence components such as Services, Connections, and Applications can also monitored.

Note:

To provision new Coherence nodes, start, and stop nodes, the Management Agent must be installed on all the hosts on which the nodes are running. For more details on provisioning Coherence nodes, see the Enterprise Manager Lifecycle Management Guide.

Prerequisites

Before you discover a Coherence cluster, you must have completed the following tasks:

Created a Coherence cluster with one JMX management node and one or more other nodes.
Started the JMX management node with the necessary parameters as defined in Creating and Starting a JMX Management Node.
Started the other nodes with the necessary parameters as defined in Configuring All Other Nodes.

To discover an already running Coherence cluster, follow these steps:

Log in to Enterprise Manager as a user with the Add Target privilege.
From the Targets menu, select Middleware. You will see a list of Middleware targets.

Note:

Alternatively, you can add a Coherence target from the Setup menu. From the Setup menu, select Add Target, then select Add Targets Manually. In the Add Targets Manually page, select the Add Non-Host Targets Using Guided Process option. Follow the steps in the wizard to add the Coherence target.
Select Standalone Oracle Coherence Cluster in the Add drop-down box and click Go. The Oracle Coherence Cluster: Discover Cluster, Node, and Cache Targets page is displayed.
On this page, select option Standalone Coherence cluster configured with dedicated management node or Standalone Coherence cluster configured with dynamic management mode (for coherence versions 12.2.1.x and above) to specify the connection details of the Coherence MBean Server. You can select either of these two options based on coherence versions in use. This is required to discover the Coherence cluster, node and cache targets.

If you have selected option Standalone Coherence cluster configured with dedicated management node, you can select either of the following options to provide MBean Server details:
Figure 19-4 Add Coherence Target
- Host, Port, and Service: Enter the following details:
  - Management Node Host: Select the host on which the Management Node is running.
  - JMX Remote Port: The port used for the JMX RMI connection. If you are using the MBean connector for Coherence MBeans, specify the tangosol.coherence.management.remote.connectionport property.
    
    Note:
    
    It is recommended that you use the com.sun.management.jmxremote.port property.
  - Service Name: The service name used for the connection. The default is jmxrmi.
- JMX Service URL: Service URL that will be used for the connection. If you enter the URL, the values specified in the Machine Name, Port, Communication Protocol, and ServiceName fields will be ignored. For example, service:jmx:rmi://localhost:3000/jndi/rmi://localhost:9000/server. For more details on the URL format, refer to http://java.sun.com/j2se/1.5.0/docs/api/javax/management/remote/JMXServiceURL.html
  
  You may need to specify the Service URL only in complex cases like when the RMI registry and the MBean Server ports are different. It is recommended that you use the Machine Name and Port option for the MBean server connection.
If you have selected option Standalone Coherence cluster configured with dynamic management mode (12.2.1.x and above), you must enter the following details:.

Figure 19-5 Add Coherence Target

Cluster Port: In dynamic management mode any coherence node configured with dynamic managed mode can be a Management Node. Hence, you must enter coherence Cluster Port to discover Coherence target instead of JMX port.

Host Name: You must enter the host name where nodes with dynamic management mode are running. You can see a list of host names on which coherence nodes with dynamic management mode are running. If the current management node is down then the next management node is discovered using these host names.

For more information, see Specifying a Cluster's Multicast Address and Port in the Oracle® Fusion Middleware Developing Applications with Oracle Coherence guide.
MBean Server Credentials: If JMX authentication is used, specify the username and password required to access the MBean Server.
Select the Management Agent that will be used to monitor the Coherence target.
Select the Do not discover Coherence Caches checkbox to skip Coherence cache targets discovery and click Continue.
If this checkbox is selected, new Coherence cache targets will not be discovered. It is recommended to skip cache discovery for clusters with large number (over 1000) of caches.
The details of the discovered targets are displayed. Click Add Targets to add these targets to Enterprise Manager.

Note:

To automatically discover a new node or target in Enterprise Manager, you must refresh the cluster as described in Refreshing a Cluster.

Refreshing a Cluster

You can manually synchronize the cluster targets with the running Coherence cluster. Click Refresh Cluster from the Oracle Coherence Cluster menu. A message indicating that new Coherence nodes and caches that have been discovered will be added as Enterprise Manager targets is displayed. Nodes are updated if there are any changes to their attributes.

Click Continue to refresh the cluster. This ensures that the latest changes are applied.

Figure 19-6 Refresh Cluster

Click Close. The list of nodes and caches that can be added are displayed.

If you want to remove already discovered caches, select Do not discover Coherence Caches checkbox, and then select Delete Existing Coherence Cache Targets .

Click Add Targets to add the targets to the cluster.

Note:

Decommissioned nodes and caches will not be removed during the Refresh process. You must remove them manually.

Managing Mis-configured Nodes

While discovering a Coherence cluster, all nodes must be started with the proper guidelines as described in Configuring a Coherence Cluster.

If a node is improperly configured or has been started without the necessary guidelines, it will be categorized as a mis-configured node and will not be a part of the newly discovered cluster. During discovery, if any improperly configured nodes are present in the cluster, you will see the following screen:

Figure 19-7 Mis-Configured Nodes

This indicates that there are some improperly configured nodes in the cluster. Click Close. The following page is displayed.

Figure 19-8 Mis-configured Nodes II

The list of improperly configured nodes along with the reasons for their failure is listed in this page. You can either choose to cancel the discovery process and fix these nodes or continue with the discovery with the properly configured nodes.

If you wish to continue with the discovery process, follow the steps listed in Discovering Coherence Targets.

If you click Cancel, the discovery process is aborted and the cluster is not refreshed. If mis-configured nodes are found during the Refresh process, they must be fixed before you can run the Refresh operation again. For more information, see Refreshing a Cluster and then discover the cluster.

Discovering a Managed Coherence Cluster

You can discover a managed Coherence cluster while discovering an Oracle Fusion Middleware / WebLogic Domain by following these steps:

Login to Enterprise Manager as a user with the Add Target privilege.
From the Targets menu, select Middleware. You will see a list of Middleware targets.
Select Oracle Fusion Middleware / WebLogic Domain from the Add drop down menu and click Go.

Figure 19-9 Add Oracle Fusion Middleware / WebLogic Domain: Find Targets
Enter the following details
- Administration Server Host: Enter the host name on which the Administration Server is installed.
- Port: Enter the WebLogic Administration Server port.
- Username and Password: Enter the user name and password for the WebLogic Administration Server.
- Agent: Enter The host name for a Management Agent that will be used to discover the Fusion Middleware targets.
Click Continue. You will see a window indicating that the targets are being discovered. Click Close. Any Coherence clusters that are present in the WebLogic Domain will listed.

Figure 19-10 Targets and Agents Assignments
Click Add Targets to add these targets to Enterprise Manager and click OK to return to the Middleware page.

For more details on Oracle Fusion Middleware / WebLogic Domain discovery, see the Oracle Fusion Middleware documentation.

Note:
If the management node of a 12.2.1 managed cluster is restarted, you must manually refresh the WebLogic Domain before you can continue monitoring the cluster.

Enabling the Management Pack

Note:

For managed Coherence clusters, you must enable the Oracle Cloud Management Pack for Oracle Fusion Middleware.

You must enable the Management Pack for Oracle Coherence if you want to use any custom features. If the management pack is not enabled, you can access only the Home pages and base platform features. To enable the Management Pack, do the following:

From the Setup menu, select Management Packs, then select Management Pack Access.
Select Oracle Coherence in the Search drop-down list and click Go.
All the Coherence targets being monitored are displayed. Check the Pack Access Agreed check box for the Coherence target and click Apply to enable the Management Pack.

Note:

Apart from enabling the Management Pack, you must grant VIEW privileges to all users on the Management Agent that is monitoring the Coherence targets. This ensures that all targets being monitored by the Management Agent are visible to the user.