1. Cloud Control Middleware Management Guide
  2. Using JVM Diagnostics
  3. Using JVM Diagnostics

17 Using JVM Diagnostics

This section describes the tasks you can perform by using JVM Diagnostics.

In particular, it contains the following:

Setting Up JVM Diagnostics

Follow these steps to set up and configure JVM Diagnostics:

  1. From the Setup menu, select Middleware Management, then select Engines and Agents. A list of RUEI/JVMD Engines are listed. Under the JVM Diagnostics Engines row, the following details are displayed when all the columns are selected:

    • Name: The name assigned to the JVM Diagnostics Engine. This ID identifies the JVM Diagnostics Engine in all the processes.

    • Type: JVM Diagnostics Engine (By default, this column in not listed).

    • Host: Machine on which the JVM Diagnostics Engine has been deployed.

    • Port: HTTP port of the server on which the JVM Diagnostics Engine has been deployed.

    • SSL Port: SSL Port of the server on which the JVM Diagnostics Engine has been deployed.

    • Availability: Percentage of time when the engine has been available.

    • Status: Status of the JVM Diagnostics Engine. Options are Active, Inactive, or n/a (not available).

    • Server: Server on which the engine is located.

    • Version: Build version of this JVM Diagnostics Engine.

  2. Select the JVM Diagnostics Engines row and click Configure to configure the JVM Diagnostics Engine parameters, JVMS and Pools, databases, and heap loader. The following tabs are displayed:

Note:

JVMD Load Balancers information is also displayed on the Setup page. The table includes Load Balancer URL, its status, and a list of engines associated with it.

Configuring the JVM Diagnostics Engine

You can configure the JVM Diagnostics Engine by defining engine parameters and advanced settings. You can then create new idle thread rules and system call rules. Operations on existing rules include importing and exporting a rules, as well as deleting a rule.

  1. From the Setup menu, select Middleware Management, then select Engines and Agents. Select the JVM Diagnostics Engine row in the RUEI/JVMD Engines table and click Configure.
  2. Click the JVMD Configuration tab.
  3. You can modify the following details in the JVMD Engine Parameters region.

    • JVMD Engine Log Level: The log level for console diagnostics messages. Log levels 1 to 5 are supported where:

      • ERROR–1

      • WARN–2 (warning)

      • INFO–3 (information)

      • DEBUG–4

      • TRACE–5

      The default log level is INFO–3.

    • Cross Tier Log Level: The log level for cross-tier diagnostic messages. Log levels 1 to 5 are supported where:

      • ERROR–1

      • WARN–2 (warning)

      • INFO–3 (information)

      • DEBUG–4

      • TRACE–5

      The default log level is INFO–3.

    • Agent Request Timeout (secs): The number of seconds that the JVM Diagnostics Engine waits for the JVM Diagnostics Agent to respond. You can increase this value if the monitored JVMs are extremely busy and the console times out and disconnects while waiting for a response.

    • Enable Monitoring: Select his check box to start or stop monitoring.

  4. Advanced Settings

    • Purge Detail Data Older Than (hours): The period for which the detailed monitoring samples should be retained.

    • Thread Stack Repository Insertion Rate (%): Enter a number between 1 and 100. The thread stacks will be stored in the repository at the specified rate.

    • System Sample Interval (secs): The frequency at which system details (cumulative CPU counters, heap size, and number of garbage collections (GCs)) should be collected in monitoring.

    • Purge Aggregated Data Older Than (days): The period for which the aggregated monitoring samples should be retained.

      Note:

      This field is not applicable to the JVMTI (level 0) optimization.

    Click Save to save the parameters.

  5. In the Thread Rules region, you can define the following:

    • Idle Thread Rules: Mark a thread as idle by adding it to an Idle Thread Rule. All threads that have been marked as idle will not be monitored. Click New Rule to create a new Idle Thread Rule and specify the idle thread rule information.

      • Rule Type: The Rule Type can be:

        Monitor (Waiting on Lock): Select this type if you want to ignore threads that are locked with a lock of the specified name.

        Current Call: Select this type if you want to ignore all threads that are making a call to the selected function (class + method).You can specify a wild card, for example, if you specify weblogic.servlet.*, all the threads that meet this criteria will be ignored.

        Note: The Current call of the stack is the first frame of the stack, traversing from top to bottom, such that the function (class + method) is not a System call. System calls are assumed to be the calls which are not relevant to the user application like java.*, and so on.

        Thread Name: Select this type if you want to ignore threads with a particular name.

      • Rule Value: The Rule Value should contain the fully qualified class name, method, followed by the class+method.

        An example of a Current Call is weblogic.socket.PosixSocketMuxer->processSockets

        An example of a Monitor (Waiting on Lock) is weblogic.socket.PosiSocketMuxer$1

        All threads that meet the criteria specified in the Idle Thread Rule will not appear in the View Active Threads screen.

      • Global Rule: Select this check box to apply the idle thread rule to all JVM Pool targets. If this box is unchecked, you must select one or more JVM Pools for which this rule will be applicable.

      All threads that meet the criteria specified in the Idle Thread Rule will not appear in the View Active Threads screen.

    • System Call Rules: Mark a method as a system call by adding it to the System Call Rules. System calls are assumed to be the calls which are not relevant to the user application like java.*, and so on. Click New Rule to create a new system call and specify the matching pattern such as sun.*, java.*, and so on.

      All methods that match the rules listed in the System Call Rules table are identified as system calls.

Configuring JVMs and JVM Pools

You can group sets of JVMs into JVM pools that provide monitoring information across all related JVMs in a single view. You can view all the JVM pools in the WebLogic Domain, create a new JVM pool, and edit existing JVM pools.

  1. From the Setup menu, select Middleware Management, then select Engines and Agents. Select the JVM Diagnostics Engine row in the RUEI/JVMD Engines table and click Configure.

  2. Click the JVMs and Pools tab. The list of all available JVM pools is displayed. For each pool, you can set the Poll Enabled flag and specify the Poll Interval. If the Polling Enabled flag is set to Yes, JVMs belonging to this pool will be polled for active requests periodically based on the Poll Interval.

  3. Click Create Pool to create a new pool.

    1. In the Create New JVM Pool dialog box, enter the name and description of the JVM pool.

    2. In the Poll Interval field, specify the sample interval for JVMs belonging to this pool when monitoring (polling) is enabled.

    3. Check the Poll Enabled check box to poll the JVMs belonging to this pool.

    4. Click Create to save the JVM Pool information.

  4. To delete a pool, highlight the pool click Remove.

  5. Select a JVM Pool or a JVM and click Details to view additional details about the JVM Pool or JVM.

  6. Click Downloads to download JVM Diagnostics components. You can download the following components:

    • JVMD Agent: Contains JVM Diagnostics Agent binaries for all supported platforms.

    • LoadHeap: Contains scripts to upload heap snapshots to the repository.

    • Load JFR: Contains scripts to upload JFR snapshots to the repository. Use JMC (Java Mission Control) to download and analyze the JFR snapshot.

  7. Select a JVM Pool or a JVM and click Configure.

    For JVM Pools, see Configuring a JVM Pool.

    For JVMs, see Configuring a JVM.

Registering Databases

Follow these steps to register databases:

  1. From the Setup menu, select Middleware Management, then select Engines and Agents. Select the JVM Diagnostics Engine row in the RUEI/JVMD Engines table and click Configure.
  2. Click the Register Databases tab. The list of registered databases is displayed. The database name, host, Oracle SID for the monitored database, and listener port number is displayed.
  3. You can do the following:

    • Add a Database Instance: From the Add menu, select Database Instance to register a single instance or Oracle Real Application Cluster (RAC) database target.

    • Add a Custom Database: From the Add menu, select Custom Database to register an external database target. Specify the Name, Host, Port, SID, Instance ID, Service, User name, Password, and Confirm Password, and click Test Connection to validate the database details. After the validation, click OK to register the database.

    • Remove: Select a database from list and click Remove to remove a registered database.

    • Edit: You cannot edit a Database Instance. Only custom databases can be edited. Select a custom database from the list and click Edit.

    • Manage DB URL: Use this option to establish cross tier correlation between JVM Diagnostics and Database Diagnostics. Select a database and click Manage DB URL. In the Associate / Disassociate Database URL(s) to the Database, select a Database URL and click Add and specify the URL of the database to be associated.

      Note:

      The DB User must be the same as the user running the application that is being monitored and must have select privileges on the GV_$SESSION, GV_$SESSION_WAIT, GV_$PROCESS, GV_$SQLTEXT, GV_$SQLAREA, GV_$LOCK, and GV_$LATCHNAME fixed views in the target database.

      To grant select privileges to a user such as jvmadmin, enter a command as follows:

      SQL> grant select on SYS.GV_$LATCHNAME to jvmadmin

      Multiple registrations may be necessary for a single database agent if different database users are running multiple applications.

    • Export: This option provides the information in a spreadsheet format. You can either open the spreadsheet or save it for future use.

  4. After the database has been registered, the JVM Diagnostics Engine will start monitoring the cross-tier JVM calls between applications being monitored for a particular JVM and the underlying database.
  5. Click Downloads to manually download the various binaries such as JVM Diagnostics Agent, Load Heap, Load JFR zip, and deploy them. You can download the following:

    • JVM Diagnostics Agent WAR File: The JVM Diagnostics Agent Parameters web.xml parameters window is displayed. From the Available Managers drop-down, you can select entries that are in the format <host>:<port> - for normal communication, <host>:<port>(secure communication) for communication over the SSL Port or you can select Other. If you select Other, you need to specify the Manager IP Address and the Manager Port to which the JVM Diagnostics Agent is connecting to. While downloading the JVM Diagnostics Agent, you can modify the following parameters:

      • Tuning Timeouts Parameters: You can modify the Connection Retry Time, GC Wait Timeout, Long Request Timeout, and Idle Agent Timeout.

      • Target Association Parameters: If you select WebLogic Server, you can specify the Target Name, and Pool Name. If you select Other Server, you can specify the Group ID Property and Pool Name.

      • Logging Parameters: You can modify the Agent Log Level.

      • Optimization Level: You can modify the Optimization Level.

    • Load Heap: The loadheap.zip is saved to a specified location.

    • Load JFR: Contains scripts to upload JFR snapshots to the repository. Use JMC (Java Mission Control) to download and analyze the JFR snapshot.

Configuring the Heap Analysis Hosts

Note:

The analysis and load heap steps have significant memory requirement on the Heap Analysis Host. Ensure you have a 64-bit JVM and sufficient free memory on the Heap Analysis Host.

To configure the heap analysis hosts, follow these steps:

  1. From the Setup menu, select Middleware Management, then select Engines and Agents. Select the JVM Diagnostics Engine row in the RUEI/JVMD Engines table and click Configure.
  2. Click the Heap Analysis Hosts tab. The Configure Heap Analysis Hosts page appears.
  3. To configure a heap analysis host, click Add and enter the following details:

    • Alias: Enter an alias for the host.

    • Heap Analysis Host: The heap analysis host on which the Management Agent has been deployed.

  4. Click Save.

Viewing Registered JVMs and Managers

Follow these steps to view a list of registered JVMs and JVM Managers:

  1. From the Setup menu, select Middleware Management, then select Engines And Agents.
  2. The list of registered JVM Diagnostics Engines are displayed.

    • Name: The name assigned to the JVM Diagnostics Engine. This ID identifies the JVM Diagnostics Engine in all the processes.

    • Type: The type of engine, in this case, JVM Diagnostics Engine. (By default, this column is not listed.)

    • Host: The machine on which the JVM Diagnostics Engine has been deployed.

    • Port: HTTP port of the server on which the JVM Diagnostics Engine has been deployed.

    • SSL Port: SSL Port of the server on which the JVM Diagnostics Engine has been deployed.

    • Availability (%): Percentage of time when the engine has been available,

    • Status: Status of the JVM Diagnostics Engine. Options are Active, Inactive, or n/a (not available).

    • Server: Server on which the engine is located.

    • Version: Build version of this JVM Diagnostics Engine.

    Highlight the JVM Diagnostics Engines row and click Configure to configure the JVM Diagnostics Engine parameters, JVMS and Pools, databases, and Heap Analysis hosts.

    • JVMD Configuration

    • JVMs and Pools

    • Register Databases

    • Heap Analysis Hosts

    • Hybrid Cloud Gateways Configuration

Accessing the JVM Diagnostics Pages

From the Targets menu, select Middleware, and click on a Java Virtual Machine Pool or Java Virtual Machine target. The Home page for the target is displayed.

To start using JVM Diagnostics, select the appropriate option from the Java Virtual Machine Pool menu.

You can also access the JVM Diagnostics pages from the WebLogic Server, WebLogic Domain, JBoss Server, or Cluster target Home pages. To do so, click on a target to navigate to the Home page. From the Target menu, select Diagnostics, then select the appropriate JVM Diagnostics menu option.

Managing JVM Pools

You can group sets of JVMs into JVM pools that provide monitoring information across all related JVMs in a single view. You can monitor all the JVMs in a pool, view historical and real time data for the JVM pool, manage threads and heap snapshots, create a new pool, and edit an existing JVM pool. JVMs and JVM Pools are now targets in Enterprise Manager. You can do the following:

Viewing the Java Virtual Machine Pool Home Page

The Java Virtual Machine Pool Home page shows the summary and configuration information of all the JVMs in the JVM pool.

It shows the following details:

  • Summary: Shows whether polling is enabled and the Polling Interval. It also shows the number of incidents and the number of configuration changes. Click the incident number to drill down to the Incident Manager page

  • Availability: This region shows the availability status of the members in the JVM Pool. Click on a Member link to drill down JVM Home Page.

  • JVM Activity (Last 15 Minutes): This region shows the active threads for each JVM in the pool. You can also select the following JVM Activity graphs: Active Thread States, Memory Utilization, CPU Utilization, GC Overhead, and Response and Load.

  • Overview (Last 15 Minutes): This region shows the status for the last 15 minutes for each JVM in the pool. The current activity of the JVM including CPU usage, memory, average number of threads waiting for a database response, network response, or average number of threads waiting for synchronization lock, idle threads, and configuration changes are displayed. Additional information includes: Target Status, Diagnostics Findings, GC Overhead %, Host CPU Usage. MAX JVM Heap Used %, Major GC Count, Minor GC Count, Major GC Time (ms), Minor GC Time (ms), Host, OS, Vendor, JVM Version, Min Heap Size (MB), Max Heap Size (MB), Open File Descriptor (%), Swap Space (%), Host Memory (%), Context Switch (per sec), and OSR.

    If JVMs displayed are present in different WebLogic domains, you can view the WebLogic Domain and the host on which the JVM is running. Click on the JVM link to drill down to the JVM Home page.

  • Top Requests (Last 15 Minutes): This region shows the top requests for each JVM in the pool. Statistics include: JVM time, JVM CPU, thread allocation, count, maximum duration of each request, the average duration of each request, throughput (per minute) and minimum duration (ms).

Promoting JVM Diagnostics Events to Incidents

An event is a notable occurrence detected by Enterprise Manager that is related to target, job, monitoring template at a particular point in time, which may indicate normal or problematic behavior. Example for events – database target going down, performance threshold violation based on metrics, unauthorized change in the application configuration file changes, failure in job execution, and so on.

An incident is an event or set of closely correlated events that represent an observed issue requiring resolution, through (manual or automated) immediate action or root-cause problem resolution.

By default JVM Diagnostics events are not promoted to incidents and will not appear in the JVM Pool or JVM Home page. To promote events to incidents, follow these steps:

  1. From the Setup menu, select Incidents, then select Incident Rules.
  2. Click Create Rule Set. In the Create Rule Set page, in the Targets region, select the All Targets of types option. Select Java Virtual Machine and Java Virtual Machine Pool target types.
  3. Click Create in the Rules region and in the Select Type of Rule to Create window, choose Incoming events and updates to events option and click Continue. The Create New Rule: Select Events page appears. In the Type drop down list, select JVM Diagnostics Threshold Violation.
  4. Then select Specific events of type JVM Diagnostics Threshold Violation.
  5. Click Add. The JVM Diagnostics Threshold Violation Rule window appears. Select the JVM Diagnostics metrics that will trigger threshold violation events. These events will be promoted to incidents. Click OK. Click Next, review the rules, and click Continue to save the rule. All events that match the criteria will be promoted to incidents and will appear in the JVM Diagnostics Pool Home page.

Viewing the JVM Pool Live Thread Analysis Page

This page shows the real-time data for all the JVMs in the selected pool. This data is useful in analyzing the various active and idle threads on the JVM. During analysis you can drill down from the thread level to methods used in the thread to local variables that are part of the method.

From the Targets menu, select Middleware, and then click on a Java Virtual Machine Pool target. Select the Live Thread Analysis option from the Java Virtual Machine Pool menu.

This page shows the following:

  • JVMs: This table shows the list of JVMs and the current status of each JVM. The current activity of the JVM including CPU usage, memory, number of threads waiting for a database response, number of threads waiting for synchronization lock, and other details are displayed

  • JVM Threads: This table shows a list of all the threads running in the selected JVM. For each thread, the following details are displayed:

    • Thread Name: Name of the thread. Click on the link to view the JVM Stack.

    • Request: Application request being processed by the thread.

    • OS PID: Operating system and Thread IDs for this thread.

    • Current Call: Lowest user method being executed by the thread.

    • File Name: Name of the file that contains the class and method for the current call.

    • Line: Line number in the method currently being executed.

    • State: The current state of the thread. This can be DB Wait, RMI Wait, or Network Wait.

      If the ADP or DMS is configured, the Request Name and Request Age values are displayed.

      If a thread is in the DB Wait State, the Waiting On column displays the name of the database the thread is waiting on, and the time the thread has to wait is displayed in the Waiting Time column.

      Click the link to view the database diagnostics details. See Performing Cross Tier Analysis for more details.You can track database issues and determine the application request responsible for the database activity. You can also view the complete call stack including the method and line number information.

      Note:

      To view the database diagnostics details, ensure that:

      • The JVM Diagnostics Agent is running on the JVM that initiated the request.

      • The monitored database must be registered by the JVM Diagnostics Engine.

    • Additional columns include: App Name, Module Name, Work Manager, Frames Count, Is Stuck, Is Hogger, Read Characters, Write Characters, Blocked Count, Blocked Time (ms), Wait Count, Waited Time (sec), IO File Name, OS Thread ID, Age (ms), Waiting Time (sec), Lock Held, ECID, RID, User, Session, and Is Idle.

    You can do the following:

    • Export: Select a thread and click Export to export the thread details along with thread stacks information to an Excel file.

    • Search/Filter: To minimize the number of reported rows, in the Search field select the column name and then provide a filter. For example, select Thread Name then type the word job. The search reports on threads that include the word job.

    • Show Idle Threads: Select this check box to list all the Idle Threads in the JVM Threads table.

    • Detach: Select a thread and click Detach to view the table in a separate window.

    • Take Snapshot of Selected Thread: Select a thread, and from the Action menu, select Take Snapshot of Selected Thread. The Thread Snapshot page is displayed. You can configure the snapshot settings and click Take Thread Snapshot. A snapshot file with details of the selected thread is generated. From the Java Virtual Machine Pool menu, select Thread Snapshots to view additional details.

    • Take Snapshot of Active Threads: This option allows you to take a snapshot of all the active threads. From the Action menu, select Take Snapshot of Active Threads, the Thread Snapshot page is displayed. You can configure the snapshot settings and click Take Thread Snapshot. A snapshot file with details of all the active threads is generated. From the Java Virtual Machine Pool menu, select Thread Snapshots to view additional details.

    • View Thread History: Select a thread and from the Action menu, select View Thread History. The historical data for the thread for the selected time interval is displayed on the Java Workload Explorer page.

    • Mark Idle: Select a thread and from the Action menu, select Mark Idle. The selected thread will be marked as Idle based on the Idle Thread Rule and will no longer be collected in the monitoring data. Marking a thread idle in JVMD will not affect the OS or JVM thread management.

    • Mark Active: Select an Idle thread and from the Action menu, select Mark Active to change the status to Active.

    • Mark System Call: Apart from the threads defined as System Calls in the JVMD Configuration page (see Configuring the JVM Diagnostics Engine), you can mark specific threads as system calls so that JVMD will not consider the marked method as a user call method.

      Select a thread from the JVM Threads table. From the Action menu, select Mark System Call to mark this thread as a System Call. All user calls that are marked in this manner will now be treated as System Calls. If you selected a marked call and click Unmark System Call, the thread will now be treated as a User Call.

  • Thread Info: This section shows the detailed information for a selected thread. Details of thread including Current Call, Request, ECID, State, Waiting On, and Wait Request are displayed. If the thread is in the DB Wait State, click the link to drill down to the Database Home page

  • Thread Stack: The Thread Stack table shows the details of the selected thread such as:

    • Class Method: The class and method for the stack frame. Click the link to view the method locals.

    • File: The file where the class is defined.

    • Line: Current execution point in the method. If a method is inlined or native, the line number might not be available.

    You can do the following:

    • Browse Local Objects: Select a method from the table and click Browse Local Objects. A popup window is displayed which shows the local variables, objects, their classes, and values.

    • Export: You can export the details of a selected thread to a file by clicking Export. You are prompted to specify the directory in which the file is to be stored. Enter the path and click Save to save file in .csv format.

  • Auto Refresh: You can refresh the data that is displayed by specifying the Auto Refresh period.

You can refresh the data that is displayed by specifying the Auto Refresh period.

Configuring a JVM Pool

From the Targets menu, select Middleware, and then click on a Java Virtual Machine Pool target. Select the Configure JVM Pool option from the Java Virtual Machine Pool menu. You can do the following:

  • Modify the JVM pool details. You can enable or disable monitoring of pools or change their polling intervals by updating the pool properties. Click Save to save the changes.

  • Configure the JVM pool thresholds. See Updating Pool Thresholds.

Updating Pool Thresholds

Follow these steps to edit the pool thresholds on the Edit JVM Pool Information page:

  1. In the Edit JVM Pool Threshold Values region, the following details are displayed:

    • Level: Thresholds violations can have a level of R (red) or Y (yellow).

    • Metric: The attribute or metric that is being monitored.

    • Threshold: The Critical and Warning threshold for the metric. A violation occurs when the threshold is exceeded after a minimum number of samples have been monitored.

  2. Click Add to add a corrective action. Select a corrective action from the list and click OK. You can select:

    • No Action: No corrective action is defined.

    • Trace Dump: Select this option to trace a particular thread, or all active threads in response to a threshold violation. You can define the following parameters:

      • Poll Interval (ms): Interval after which snapshot should be repeated.

      • Poll Duration: (sec) Duration for which the snapshot should be taken.

      • Description: Describe the purpose of the trace dump. Include information pertinent to other users.

      • Thread Details: You can specify if the thread details need be included in the snapshot.

      • Try Changing Threads: Sometimes the stack associated with the thread may change rapidly which makes taking the snapshot difficult. If you select this parameter, you can suspend the thread and take the snapshot.

      • Include Network Waits: Specify if network wait threads need to be included in the snapshot.

      • All Threads: Specify if all threads (active and idle) must be included in the snapshot.

    • Heap Dump: Select this option to generate a heap dump in response to a threshold violation. The Heap Snapshot Type can be:

      • TXT: Text (txt) for analysis in JVM Diagnostics.

      • HPROF: Binary format for analysis with external tools.

      If a corrective action (trace dump or heap dump) is generated due to a threshold violation, the trace or heap dump files are displayed in the Event Details page. See Viewing JVM Diagnostics Threshold Violations.

    • JFR Snapshot: Select this option to generate a dump of the JFR.

      JFR snapshot creation is supported for Java JVM and for Oracle JDK 1.7.0_04 onwards. For Oracle JDK 1.7.0_04 onwards but prior to JDK 1.8.0_40, JVM process should be run with the following java options '-XX:+UnlockCommercialFeatures -XX:+FlightRecorder'. These java options are not required for JDK 1.8.0_40 onwards.

    • Class Histogram: Select this option to generate a dump of the class histogram.

    • Diagnostic Snapshot: Select this option to generate a diagnostic dump.

      Select the Duration in Minutes to be used in this snapshot.

  3. Click Save to save the threshold values.

Removing a JVM Pool

You can remove a JVM Pool from the following:

  • Middleware page: Highlight the JVM Pool and click Remove.

  • JVM Pool home page: From the Java Virtual Machine Pool menu, select Target Setup, then click Remove Target.

You will see a warning message that displays the name of the target being deleted and that when a pool is deleted, all the JVM targets in the pool are also displayed. Click Yes to delete the JVM Pool or No to return to the JVM Pool Home page.

Adding a JVM Pool to a Group

From the JVM Pool home page, select the Java Virtual Machine Pool menu. Select Target Setup, then click Add to Group...

Select this option to add the JVM Pool to one or more groups. A pop-up window appears with a list of groups on which you have Operator privileges. Select one or more groups and click Add to add the target to the group.

Managing JVMs

You can monitor a specific JVM in a pool, view historical and real time data, and so on. You can do the following:

Viewing the JVM Home Page

The JVM Home page shows the summary and configuration information of all the JVMs in the JVM pool. Follow these steps to view the JVM Home page:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target.
  2. The JVM Home page with the following details is displayed.

    • Summary: Shows details of the JVM such as the availability status of the JVM, heap size, WebLogic Server it belongs to, and composite applications. The region also includes the number of open incidents that have occurred and the number of configuration changes. Click the number of incidents to drill down to the Incident Manager page.

    • JVM Activity (Last 15 Minutes): The number of active threads in the JVM in the last 15 minutes. Click on a thread to see the detail of the thread. You can also select the following JVM Activity graphs: Active Thread States, Memory Utilization, CPU Utilization, GC Overhead, and Response and Load.

    • Overview (Last 15 Minutes): Shows the state of the various threads in the JVM in the color-coded columns. This region can be added using page customization.

      The current activity of the JVM including Target Status, Threads (CPU, DB Wait, Lock, Network Wait, IO Wait, RMI Wait), JVM Time (sec), Diagnostic Findings, Resource (%), Host CPU (%), JVM DPU(%), JVM Heap (%), Max JVM Heap (%), GC Overhead (%), Major GC Count, Minor GC Count, Major GC Time (ms), Minor GC Time (ms), Host, OS, Vendor, Version, Container Name, Container Type, Min Heap Size (MB), Max Heap Size (MB), Open File Descriptors (%), Swap Space (%), Host Memory (%), Context Switch (per sec), and OSR are displayed.

    • Top Requests (Last 15 Minutes): Shows: Shows the top requests for the JVM. Statistics include: JVM time, JVM CPU, thread allocation, count, maximum duration of each request, the average duration of each request, throughput (per minute), and minimum duration (ms).

Viewing the JVM Diagnostics Performance Summary

You can view the performance metrics (system and active threads) for a JVM target on the Performance Summary page. A set of charts is displayed on this page for the JVM target. To view the JVM performance metrics, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target.
  2. Select the Performance Summary option from the Java Virtual Machine menu. The following page appears:
  3. A set of default charts that show the values of the JVM performance metrics over a period of time is displayed. Review the metrics for any periods of time where the Warning or Critical Thresholds were reached.

    If any of the metrics exceed the Warning Thresholds or Critical Thresholds, it could indicate memory is a factor in the JVM performance and availability. It could mean there is a memory leak or that the JVM heap configuration is too small for the application load. If the heap configuration is correct, you must review the real time heap data. You can then create a snapshot that can be examined for leaks. See Taking a Heap Snapshot for details.

  4. Click the Show Metric Palette button. The metric palette has a folder for the current target (JVM) and the related targets. You can add or remove metric charts. Leaf nodes act as check boxes. Clicking a leaf node causes a chart to be added. Clicking it off removes the metric. Dragging a leaf node from the palette to a chart or legend adds the metric to that chart.

Note:

Agent Diagnostics Charts (raw data), JVM and Host States (raw data), JVM Heap Memory Usage (raw data), and Thread (Active) States (raw data) metric groups are deprecated from JAVA Virtual Machine Pools.

Viewing the JVM Live Thread Analysis Page

This page shows the real-time data for a selected JVM. This data is useful in analyzing the various active and idle threads on the JVM. During analysis you can drill down from the thread level to methods used in the thread, to local variables that are part of the method. To view this page:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target.
  2. Click the Live Thread Analysis link at the top of the page or select the Live Thread Analysis option from the Java Virtual Machine menu.

This page shows the following:

  • JVMs: This table shows the list of JVMs and the current status of each JVM. The current activity of the JVM including CPU usage, memory, number of threads waiting for a database response, number of threads waiting for synchronization lock, and other details are displayed. JVM Process statistics include: CPU (%), Memory (%), Context Switch (per sec), and Open File Descriptors (%). Threads statistics include: CPU, DB Wait, Lock, Network Wait, IO Wait, RMI Wait, and Idle Threads.

  • JVM Threads: This table shows a list of all the threads running in the JVM. Click on a thread to view the thread details. To view all the available columns, from the View menu, select Columns, then select Show All.

    Thread details include:

    • Thread Name: Name of the thread. Click on the link to view the JVM Stack.

    • Request: Application request being processed by the thread.

    • OS PID: Operating system and Thread IDs for this thread.

    • Current Call: Lowest user method being executed by the thread.

    • File Name: Name of the file that contains the class and method for the current call.

    • Line: Line number in the method currently being executed.

    • State: The current state of the thread. This can be DB Wait, RMI Wait, or Network Wait. If the thread is the DB Wait state, click on the link to view the database diagnostics details. You can track database issues and determine the application request responsible for the database activity. You can also view the complete call stack including the method and line number information.

      Note:

      To view the database diagnostics details, ensure that:

      • The JVM Diagnostics Agent is running on the JVM that initiated the request.

      • The monitored database must be registered by the JVM Diagnostics Engine.

    If a thread is in the DB Wait State, the Waiting On column displays the name of the database the thread is waiting on, and the time the thread has to wait is displayed in the Wait Time column. You can click on the link in the DB Wait column to view the database diagnostics details. This is helpful in tracking database issues and determining the application request responsible for the database activity.

    Note:

    You can view the database diagnostics details if:

    • The JVM Diagnostics Agent is running on the JVM that initiated the request.

    • The monitored database must be registered by the JVM Diagnostics Engine.

    You can perform the following actions:

    • Export: Select a thread and click Export to export the thread details along with thread stacks information to an Excel file.

    • Search/Filter: To minimize the number of reported rows, in the Search field select the column name and then provide a filter. For example, select Thread Name then type the word job. The search reports on threads that include the word job.

    • Show Idle Threads: Select this check box to list all the Idle Threads in the JVM Threads table.

    • Detach: Select a thread and click Detach to view the table in a separate window.

    • Take a Snapshot of a Selected Thread or Active Threads: Select a thread from the list and choose the Take Snapshot of a Selected Thread option from the Action menu. The Thread Snapshot page is displayed where you take a snapshot. If you select the Take Snapshot of Active Threads option, you can take a snapshot of all active threads running on this JVM. You can specify the following parameters for each snapshot:

      • Poll Interval: Interval after which snapshot should be repeated.

      • Poll Duration: Duration for which the snapshot should be taken.

      • Description: Description of the snapshot.

      • Thread Details: You can specify if the thread details need be included in the snapshot.

      • Try Changing Threads: Sometimes the stack associated with the thread may change rapidly which makes taking the snapshot difficult. If you select this parameter, you can suspend the thread and take the snapshot.

      • Include Network Waits: Specify if network wait threads need to be included in the snapshot.

      • All Threads: Specify if all threads (active and idle) must be included in the snapshot.

      • Allow Trace Interrupt: Indicate whether the trace process can be interrupted.

      A snapshot file with details of the selected thread or active threads (depending on your selection) is generated. From the Java Virtual Machine Pool menu, select Thread Snapshots to view additional details.

    • View Thread History: Select a thread and from the Action menu, select View Thread History. The historical data for the thread for the last 30 minutes is displayed.

    • Mark Idle: Select a thread from the list and from the Action menu, select Mark Idle to mark a thread as idle.

    • Mark Active: If you selected the Show Idle Threads check box, a list of idle threads is displayed. Select a thread and from the Action menu, select Mark Active to mark it as an active thread.

    • Mark System Call: Apart from the threads defined as System Calls in the JVMD Configuration page (see Configuring the JVM Diagnostics Engine), you can mark specific threads as system calls. Select a thread from the JVM Threads table. From the Action menu, select Mark System Call to mark this thread as a System Call. All user calls that are marked in this manner will now be treated as System Calls. If you selected a marked call and click Unmark System Call, the thread will now be treated as a User Call.

  • Thread Info: This section shows the detailed information for a selected thread. Details of thread including Current Call, Request, ECID, State, Waiting On, and Wait Request are displayed. If the thread is in the DB Wait State, click on the link to drill down to the Database Home page. See Performing Cross Tier Analysis.

  • Thread Stack: The Thread Stack table shows the details of the selected thread such as:

    • Class Method: The class and method for the stack frame. Click on the link to view the method locals.

    • File: The file where the class is defined.

    • Line: Current execution point in the method. If a method is inlined or native, the line number might not be available.

    You can drill down from the method level to a lower level. You can do the following:

    • Browse Local Objects: Select a method from the table and click Browse Local Objects. A popup window is displayed which shows the local variables, objects, their classes, and values.

    • Export: You can export the details of a selected thread to a file by clicking Export. You are prompted to specify the directory in which the file is to be stored. Enter the path and click Save to save the file in .csv format.

    • Mark / Unmark System Call: You can mark a selected method as a system call. Select a method from the Thread Stack table and from the Action menu, select Mark System Call. All methods marked in this manner will be treated as system calls. If you select a marked call and click Unmark System Call, the method will now be treated as a user call.

  • Auto Refresh: You can refresh the data that is displayed by specifying the Auto Refresh period.

Performing Cross Tier Analysis

You can trace any JVM activity from the JVM thread to the database. You can view cross tier correlation for live threads and historical monitored data.

Before you establish cross tier correlation, ensure that the database is an Enterprise Manager target and has been registered with JVM Diagnostics. This enables you to drill-down from JVM Diagnostics pages to Database Diagnostics pages.

Note:

If a database is not registered with JVMD, the Database JDBC details, SQL statement, SQL ID, and schema name will be collected by the JVMD agent for threads in DB Wait state.

In the case where the database is not an Enterprise Manager target, you can still register the database with JVMD as a “Custom database" which will track the database activity to this database by its name. The SQL statement and SQL ID would be fetched real-time from the database but you cannot drill-down from JVM Diagnostics pages to Database Diagnostic pages.

To register the database:

  1. From the Setup menu, select Middleware Management, then select Setup. Select the JVM Diagnostics Engine row in the RUEI/JVMD Engines table and click Configure.

  2. Click the Register Databases tab. The list of registered databases is displayed. The database name, host, Oracle SID for the monitored database, and listener port number is displayed. You will also see a column indicating “JVMD Supported DB". If this column has a value of Yes, you can proceed with the cross tier analysis. If the column has a value of Unavailable, you cannot perform cross tier analysis because the JDBC connection to the database cannot be established.

    Note:

    If cross tier correlation is not established even after registering the database with JVMD, select a database and click Manage DB URL. In the Associate / Disassociate a Registered Database field, select a Database URL and click Add and specify the URL of the database to be associated. After the URL has been associated with the registered database, the JVM Diagnostics Engine will start monitoring the cross-tier calls between JVM targets and the underlying database.

To view the cross tier correlation for live threads, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Live Thread Analysis option from the Java Virtual Machine menu.

  2. In the JVM Threads column, select a thread with a DB Wait State.

  3. The thread details are displayed in the Thread Info section. If cross tier correlation has been established, you can see SID=<value>"SERIALNUM=<value> when you hover over the State field. Click the DB Wait link to navigate to the Database Diagnostics pages.

    Note:

    If cross tier is not established, the Database Details popup is displayed which shows the host, port, SID, user, and JDBC URL for the target database. This can happen when the database is not registered with JVMD or if JVMD is unable to find the registered database corresponding to the JDBC URL of the database. For registering the database, click on the link in To view Register Database page click here and this will take you to the Register Databases tab. In the case where the database is already registered, associate the JDBC URL with a registered Database by clicking the link in To associate a Registered database to the above Database URL click here. This will open a popup that will enable you to associate the JDBC URL for the database with a registered database.

To view the cross tier correlation for historical monitored data, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the JVM Performance Diagnostics option from the Java Virtual Machine menu.
  2. The three tables Top Databases, Top SQLS, and Top DBWait Events related to the cross tier are displayed. The Top Databases table shows the top databases in which JVM or JVMs in the pool have made activities. The Top DBWait Events table shows the top DB Wait events caused by the JVM threads in the database. The Top SQLs table shows the list of SQLs sorted by the number of samples.

    Click a SQL in Top SQLs to view the list of registered databases on which the SQL was executed. “Not Defined" represents SQLs that were executed on databases that are not registered with JVMD. Click a Database in Top Databases opens a popup that shows a link to the database name, clicking on which takes to Database diagnostics page. In case the database name is “Not Defined", the popup shows the different JDBC URLs for the databases that were not registered and the corresponding number of samples for each Database. It also shows the reason as to why the cross tier was not established.

  3. Click the Database Name link to drill down to the Database Diagnostics page which shows the corresponding database activity.
  4. Click the Top SQLs and Top DBWait Events links to navigate to the SQL Details page and the ASH Viewer page of database diagnostics.

    If cross tier correlation has been established, you can view JVM Diagnostics activities for a database (drilling up from Database Diagnostics to JVM Diagnostics). Click the JVM Diagnostics link in the Performance page to drill up to the JVM Performance Diagnostics page. Data relevant to the time interval, database and other filters is displayed.

Establishing Cross-Tier Correlation in Oracle RAC Databases

Oracle Real Application Cluster (Oracle RAC) databases have a complex configuration of database instances and listeners. User applications use Oracle RAC services to connect to the database instead of SIDs that are used for single instance databases. User applications can connect to Oracle RAC listeners that are listening on different machines than the actual database instances. For cross tier correlation to be established, all the listener and database instances must be discovered targets in Enterprise Manager. Cross tier correlation can be established by using either of the following options:

  1. If you have the cluster (RAC) database discovered in your EM and want to do the Xtier correlation with JVMD, then you can go and add the target type of Cluster Database. Refer the figure below:

    Note:

    You should not select the database instance which are part of the cluster. For example, in figure given below racdb is cluster database and racdb_racdb1 and racdb_racdb2 are part of cluster racdb, so if you are using the cluster database in your weblogic data source then you should consider adding cluster database. JVMD will get the associated database instances for you automatically.

    Figure 17-1 Adding Cluster Database


    adding cluster
    In the above figure, the cluster Database racdb is added and the below figure shows both database instances racdb_racdb1 and racdb_racdb2 being added to registered databases.

    Figure 17-2 After registering cluster database


    after registering

    If you want to add only database instances which are part of cluster, you can do that but you may miss some of the mappings if the SQL execution request goes via another db instance. Hence, it is recommended to add cluster database directly if it's discovered in EM since this covers more big spectrum for Xtier mapping.

  2. If all the database instances in the Oracle RAC have not been discovered in Enterprise Manager:

    • Register the database instances with JVM Diagnostics as custom databases using SID for each instance of RAC DB and add the jdbc url (which is used to to connect to application via configured data source) in Manage DB URL.

      Figure 17-3 Manage Database URL


      manage database

      Consider the following example:

      RAC DB interface: host_rac, port:1521, ServiceName:S121, having the three db instances configured as,

      DB1--> hos1 SID:S1212 Port:1521

      DB2--> host2 SID:S1211 Port:1521

      DB3--> host3 SID:S1213 Port:1521

      And data source is configured as jdbc:oracle:thin:@//host_rac:1521/S121. Then in above scenario you must add three custom DB using their SID and for each custom DB add the jdbc url configured in the data source via Manage DB URL.

Viewing Memory Diagnostics

This page provides you the details regarding current memory pool usages and the garbage collections statistics. It also provides statistics related to the class loading and class compilations, and the means to get and save a live histogram, and view all the histograms.

Follow these steps to do a real-time analysis of the JVM heaps that have been loaded into the repository.

  1. From the Targets menu, select Middleware, then click on a JVM target.
  2. From the Java Virtual Machine menu, select Memory Diagnostics.
  3. The following tabs are available:

    • Heap Memory Usage

      These charts depict java memory pool usage.

      • JVM Heap Memory Usage

        The SunBurst chart shows the java heap structure and sizes. The Heap node in the center of the sunburst represents the heap-memory of the JVM process. When you mouse over the Heap node, it shows the size of the heap in MBs. Surrounding these nodes are other memory pools which are part of the JVM heap memory. Double clicking any pool node expands the sunburst and that pool node becomes the center of the chart and its children "Used" and "Free" are shown.

        For HotSpot JVM, there are three children of heap node: Eden Space, Survivor Space, and Old Gen (generation). Mouse over these nodes to view there sizes.

        The bar chart shows percent utilization of all the memory pools (heap and non-heap). If the pool usage is less than 75%, the color of the bar is green. If the pool usage is between 75% and 95%, the color of the bar is yellow. If the pool usage is 95% and higher, the color of the bar is red.

      • Tenure Distribution Information

        Tenure represents the age of the JVM objects that survived the number of minor collections. For example, an object with Tenure Size of 2 represents that these objects have survived two minor collections.

        The statistics that display are dependent on the version of the JVM and the configuration of the garbage collection.

      • TLAB (Thread Local Allocation Buffer) Statistics

        This region shows Thread Local Allocation Buffer (TLAB) related JVM performance counters.

        To avoid the pointer contention in the Eden Space while allocating objects, each thread is given a private memory area where it does object allocation. This private memory area is called TLAB.

      • JVM Metrics

        This region shows different metrics charts (by percentage and by value) related to JVM heap. Monitoring data from the JVM is used to prepare the correlation charts.

        Use the time selector to select the desired time period for the charts. By default, chart duration is 15 minutes.

        To view the information in table format, click Table View.

      Note:

      For JVMs running at optimization level, the following details are displayed:

      • JVM Heap Memory Usage table where the usage (in KB) in various heap spaces.

      • JVM Heap Number of Objects table which displays the number of objects in various heap spaces.

    • Garbage Collection

      These charts and tables report the number of objects that have been added to the garbage collection (gc). The type of garbage collection, that is minor or major, and the number of garbage collections of a particular type are displayed.

      • Last Garbage Collection Information

        The Last Garbage Collection Information region provides the information pertaining to the last major and minor garbage collection that occurred in the JVM including: start time, end time, duration of the last collections, garbage collector name, current garbage collection count, and number of garbage collection threads. If provided by the JVM, this tab also shows the cause of the garbage collection.

        Also included are bar charts which show the changes in pool usages after the garbage collection.

      • Garbage Collections

        The Garbage Collections region shows the cumulative statistics of major and minor collections including total collection count, total gc pause time (in milliseconds), rate of collections (invocation per minute), and the gc overhead percentage.

      • JVM Metrics

        This region shows different metrics charts (by percentage and by value) related to garbage collection. Monitoring data from the JVM is used to prepare the correlation charts.

        Use the time selector to select the desired time period for the charts. By default, chart duration is 15 minutes.

        To view the information in table format, click Table View.

    • Class Loading Data

      This tab shows data related to class loading and class compilations.

      • Class Loading Stats

        This region shows the total number of classes loaded, the total number of classes unloaded, and the total class loading time (in milliseconds) since the JVM started. It also shows total compilations (just-in-time [JIT] + on stack replacement [OSR]) done and total compilation time (in milliseconds) since the JVM started. Finally, it shows the number of objects which are pending finalization.

      • JVM Metrics

        This region shows different metrics charts related to class loading and compilations. Monitoring data from the JVM is used to prepare the correlation charts.

        Use the time selector to select the desired time period for the charts. By default, chart duration is 15 minutes.

        To view the information in table format, click Table View.

    • Class Histograms

      Use this tab to generate a live histogram, save it, and see all the saved histograms. When you click Get Live Histogram, the JVM heap is traversed and a histogram containing class name, instance count, total size and average size is generated. You can then study the histogram and see the classes whose instances are consuming the most memory.

      The JVM Class Details table provides a summary of the heap usage by different types of objects in the heap.

      • Class name: The name of the space within the JVM heap.

      • Instance: The number of heap objects for number of instances of classes in a heap space.

      • Total Size (KB): The size of the JVM heap.

      • Average Size (KB):

      • Get Live Histograms

        Click this option to generate a histogram.

      • Schedule

        Click Schedule to add the JVM Class Histogram data to the repository by scheduling a job. You can specify the schedule as Immediate or Later. If you select Later, you can specify if the job needs to be run only once or repeated at specified intervals.

      • View Saved Histograms

        Click this option to view the saved histograms in the Available Heap Snapshots page.

  4. You can do the following from any of the tabs:

    • Take Heap Snapshot

      Click Take Heap Snapshot to take a heap snapshot.

    • Export

      Click Export to export live heap data to an Excel file.

    • Save

      Click Save to save the classes in the JVM Classes table to the repository. The Save Class Histogram popup appears. Enter a name for the snapshot, and a description, and click OK.

Working with Class Histograms

A class histogram is displayed in the form of a table when the optimization level of the jamagent is 0. The histogram displays the top 300 data rows sorted by the size. You can perform various operations on class histograms. This section describes the following:

Saving a Class Histogram

To save a class histogram, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Memory Diagnostics option from the Java Virtual Machine menu.
  2. In the Memory Diagnostics page, click the Class Histograms tab. Click Get Live Histogram to get Class Histogram data. Click Save.
  3. In the Save Class Histogram window, enter a name for the snapshot and a description and click OK.
Viewing Saved Histograms

To view saved histograms, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Memory Diagnostics option from the Java Virtual Machine menu.
  2. In the Memory Diagnostics page, click the Class Histograms tab. Click View Saved Histograms. The Available Heap Snapshots page appears.
  3. Scroll down to the Available Class Histograms table to view a list of saved class histograms.
Scheduling a Histogram Job

Scheduling will allow you to insert JVM Class Histogram data into the repository by running the job at the defined time. To schedule a class histogram job, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Memory Diagnostics option from the Java Virtual Machine menu.
  2. In the Memory Diagnostics page, click the Class Histograms tab. Click Schedule. The Schedule Settings page appears.
  3. Enter a name and description for the job to be scheduled.
  4. Specify the schedule as Immediate or Later. If you select Later, you can specify if the job needs to be run only once or repeated at specified intervals.
  5. Select the frequency at which you want to repeat the job from the Repeat drop-down list.
  6. Select the option for the Grace Period. If you select the grace period, the job will remain active and run within the specified grace period.
  7. Click OK to schedule the histogram job. A confirmation window appears indicating that the job has successfully submitted.

    To view the job status, from the Enterprise menu, select Job, then select Activity. Select the Job Type as All, and Target Type as Targetless to see the histogram job.

Comparing Class Histograms

The compare functionality allows you to compare any two class histogram snapshots listed in the table. To compare class histograms, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Memory Diagnostics option from the Java Virtual Machine menu.
  2. In the Memory Diagnostics page, click the Class Histograms tab. Click View Saved Histograms. The Available Heap Snapshots page appears.
  3. Scroll down to the Available Class Histograms table to view a list of saved class histograms. Select any two class histograms and click Compare. The Compare Class Histograms page appears. The Class Name, Instance Size (size of each snapshot), and Number of Instances (for each snapshot) are displayed.
Deleting Class Histograms

To delete class histograms, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Memory Diagnostics option from the Java Virtual Machine menu.
  2. In the Memory Diagnostics page, click the Class Histograms tab. Click View Saved Histograms. The Available Heap Snapshots page appears.
  3. Scroll down to the Available Class Histograms table to view a list of saved class histograms. Select the class histogram you want to delete and click Remove. A confirmation message is displayed. Click OK to delete the class histogram.

Taking a Heap Snapshot

A heap snapshot is a snapshot of JVM memory. It shows a view of all objects in the JVM along with the references between those objects. It can be used to study memory usage patterns and detect possible memory leaks. To take a heap snapshot, follow these steps:

  1. From the Targets menu, select Middleware, then click on a JVM target. The JVM Home page is displayed.
  2. Select Memory Diagnostics from the Java Virtual Machine menu, then click Heap Memory Usage.
  3. Click Take Heap Snapshot. The Load Heap Snapshot page appears.
  4. The Load Heap Snapshot page appears. In the Heap Snapshot Formats and Analysis Options region, specify the following:

    • Heap Snapshot Type: Select the heap dump file format. This can be:

      • JVMD Format (txt)

      • HPROF Format (binary)

        Choose this option to take a heap snapshot and load it manually to repository using the loadheap script.

    • Request GC before taking heap snapshot: Click the check box if you want to delete unused objects before the heap snapshot is taken.

    • Heap Analysis Options: Select the required option from the drop down menu:

      • Load Heap Data to Repository: Select this option to take a heap snapshot and automatically load it to the repository. If you select this option, you must ensure that the following prerequisites are met. See Taking a Heap Snapshot and Loading Into the Repository.

      • Memory Leak Report: Select this option to generate a memory leak report. The memory leak report tab shows the potential memory leak sources by identifying frequent patterns in the heap graph.

      • Antipattern Report: Select this option to generate an anti-pattern report. This report shows the summary or one kind of anti-pattern issue. This option is available only if the Heap Snapshot Type is HPROF (binary).

      Note:

      Leave this field blank if you want to do a heap dump only.

      To take a heap snapshot and load it manually to the repository, select the Heap Snapshot Type and leave the Heap Analysis Options field blank.

    • Heap Snapshot Time: Schedule the heap snapshot to occur now or schedule it for a later time.

  5. Click Submit. On the resulting dialog box, click Yes to continue the heap snapshot job. You can monitor the progress of the Heap Snapshot job. The heap snapshot is generated and the file name in which it is stored is displayed. You can upload the heap snapshot and analyze it using appropriate options from the Heap Snapshots menu.

Taking a Heap Snapshot and Loading Into the Repository

Select this option to take a heap snapshot and automatically load it into the repository.

Prerequisites

  • The Management Agent must be deployed on the host machine on which the JVM target is running.

  • The Heap Loader Host is a standalone machine (with high CPU and Memory) on which the Management Agent has been deployed.

  • DB Client Home which is the location of ORACLE_HOME where sqlldr & sqllplus are present.

  • There should be sufficient disk space in the system temp directory.

  • A JVM Diagnostics DB User must have been created using the create_jvm_diagnostic_db_user script. The script is located inside loadheap.zip. You can find loadheap.zip at: 

    $MW_HOME/plugins/oracle.sysman.emas.oms.plugin_12.X.X.X.X/archives/loadheap.zip

    The script is called by loadheap.sh. If you execute the script directly, you will be asked to input the required data. There is a README.txt file inside the loadheap.zip file that provides additional information.

To take a heap snapshot and load it into the repository, follow these steps:

  1. Select the Heap Snapshot and Load Into Repository option. You can select this option if the Management Agent is running on the JVM Diagnostics Agent and the Heap Loader Host.
  2. Specify whether the heap snapshot is taken immediately or at a later date.
  3. Specify the credentials for the host on which the JVM Diagnostics Agent is running.
  4. If the Heap Loader Host has not been configured, click Add. Provide an Alias for the host and select the host (Heap Analysis Host) target on which the Management Agent is running. Click Save.
  5. If the Heap Loader Host has already been configured, the Available Heap Loaders are displayed. Select a heap loader from the list and enter the credentials for the Heap Loader host.

    Note:

    • If preferred credentials for JVM Target & Heap Loader host are set, then the Enter Credentials region will not be displayed.

    • If the Named Credentials for the JVM Diagnostics DB User is set, the Enter Credentials region will not be displayed.

  6. Click Submit to submit the heap snapshot job. A confirmation message is displayed. Click Yes to continue. The job details are displayed in the Heap Analysis Job page. Click on the link to view the job status.

Analyzing Heap Snapshots

The JVM Diagnostics memory analysis feature allows you to not only find the objects responsible for the growth but also track their reachability from the root-set. With this feature, you can find the dangling reference responsible for memory leaks. To find a memory leak, you take snapshots of the JVM heap at different points in time. Between the snapshots, your JVM and Java applications continue running at full speed with zero overhead.

A heap snapshot is a snapshot of JVM memory. Each snapshot stores information about the objects in the heap, their relationships and root-set reachability. You can load the snapshots into the repository, and compare them to see where the memory growth has occurred. Click Heap Snapshots and Class Histograms from the menu in the JVM Pool or JVM Home page. The following page appears:

This page contains the following regions:

  • Available Heap Snapshots: You can specify the Target Name and Target Type to filter the heap snapshots that are displayed. You can also specify the Heap ID in the Snap Name field to search for specific heap snapshots and display them. The following details is displayed:

    • Heap ID: The identification number for the heap snapshot.

    • Date: The date on which the heap snapshot was taken.

    • JVM Name: The server on which the JVM is running.

    • Vendor: The name of the JVM Vendor.

    • Size: The total size of the Java heap. An adequate heap size helps improve the performance of the application.

    • Used: The amount of heap that has already been used.

      Note:

      If the heap snapshot was taken in HPROF format, the value in the Size and Used fields will be 0.

    • Used(%): The percentage of heap used.

    You can do the following:

    • Click Create to take a heap snapshot. See Taking a Heap Snapshot.

    • Select a heap snapshot and click the Detail link to drill down to the Roots page. See Viewing Heap Usage by Roots.

    • Select a heap snapshot and click Load to load the heap snapshot to the repository. See Uploading Heap Snapshots.

    • Select a heap snapshot and click Reports to download heap reports to the local host. These reports must have been generated and loaded to the repository for the selected heap snapshot. You can download the Memory Leak Report and the Antipattern Report.

  • Available Class Histograms: The list of saved histograms with details such as date on which the snapshot was taken, Snap ID, Timestamp, JVM Name and Version, Description are displayed. See Working with Class Histograms.

Viewing Heap Usage by Roots

To view the heap usage by each class of root, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine or a Java Virtual Machine Pool menu.
  2. Select Heap Snapshots and Class Histograms from the Java Virtual Machine or Java Virtual Machine Pool target menu.
  3. The list of available heap snapshots is displayed. Select a heap snapshot and click Detail to view the number of objects and memory reachable from each root. Click the Roots tab to view the objects directly reachable from the root. The following details are displayed:

    • Root: The name of the root is displayed here. Click on the name to drill down to the Top 40 Objects page.

    • Objects: The total number of objects reachable from this root.

    • Total Memory: The total amount of memory reachable from this root.

    • Adjusted Memory: The adjusted memory reachable from this root. This parameter is useful in tracking the memory leak hot-spots.

  4. Click the Usage tab to view the Heap Usage by Objects.
  5. Click the Dominator Roots tab to view the total size of all objects that would be removed when garbage collection is performed on this node.
  6. Click the Memory Leak Report tab to view the Memory Leak Report.
  7. Click the Anti-Pattern Report tab to view the Anti Pattern Report.
Top 40 Objects

This page shows the top 40 objects reachable from a root. The objects are sorted in descending order by the adjustable memory reachable from the object (or the difference of the adjusted memory reachable when comparing two heaps). This view provides a lot of rich detailed information like the amount of memory used by an object, amount of memory reachable by an object (total memory used by all the children), and number of objects reachable from a given object.

  1. From the Targets menu, select Middleware, then click on a JVM or JVM Pool target.
  2. On the JVM or JVM Pool Home page, select Heap Snapshots and Class Histograms from the Java Virtual Machine/Java Virtual Machine Pool menu.
  3. The list of available heap snapshots is displayed. Select a heap snapshot and click Detail to view the number of objects and memory reachable from each root. Click the Roots tab to view the objects directly reachable from the root.
  4. Click a root to view the top 40 objects.

The following details are displayed:

  • Signature: The signature of the object.

  • Root: This is the internal root identifier.

  • Type: The type of the object which can be Klass, Instance, Method, and so on.

  • Field:

  • Space: The heap space in which the object is present.

  • Bytes: The amount of space used by the object.

  • Len: If the object is an array, the length of the array is displayed here.

  • Children: The number of descendants reachable from the object.

  • Adj (bytes): Adjusted memory reachable from this object.

  • Retained Memory: The total size of all objects that would be removed when garbage collection is performed on this node.

  • Depth: Indicates how far this object is from the root.

Heap Object Information

This page shows information about a specific object in the heap snapshot. The following details are displayed:

  • Heap Object Information

    • Gar: Indicates whether this object is garbage or reachable from the root.

    • Space: The heap space in which the object is present.

    • Type: The type of the object which can be Klass, Instance, Method, and so on.

    • Signature: The signature of the object.

    • Bytes: The amount of space used by the object.

    • Len: If the object is an array, the length of the array is displayed here.

    • Children: The number of descendants reachable from the object.

    • Adj: Adjusted memory reachable from this object.

    • Retained Memory: The total size of all objects that would be removed when garbage collection is performed on this node.

    • Depth: Indicates how far this object is from the root.

  • Roots

    • Type: The type of root which can be Klass, Instance, Method, and so on.

    • Field: If the root is a local thread, this field contains information about the thread and method.

  • Object Children

    • Gar: Indicates whether this child is garbage or reachable from the root.

    • Space: The heap space in which the child is present.

    • Type: The type of the child which can be Klass, Instance, Method, and so on.

    • Signature: The signature of the child. Click on the link to drill down to the Details page.

    • Bytes: The amount of space used by the child.

    • Len: If the child is an array, the length of the array is displayed here.

    • Children: The number of descendants reachable from the child.

    • Adj: Adjusted memory reachable from this child.

    • Retained Memory: The total size of all objects that would be removed when garbage collection is performed on this node.

    • Depth: Indicates how far this child is from the root.

  • Object Parents

    • Gar: Indicates whether this parent is garbage or reachable from the root.

    • Space: The heap space in which the parent is present.

    • Type: The type of the parent which can be Klass, Instance, Method, and so on.

    • Signature: The signature of the parent. Click on the link to drill down to the Details page.

    • Bytes: The amount of space used by the parent.

    • Len: If the parent is an array, the length of the array is displayed here.

    • Children: The number of descendants reachable from the parent.

    • Adj: Adjusted memory reachable from this parent.

    • Retained Memory: The total size of all objects that would be removed when garbage collection is performed on this node.

    • Depth: How far this parent is from the root.

Comparing Heap Snapshots

To find a memory leak, you can take snapshots of the JVM Heap at different points in time. Each snapshot stores information about the objects in the heap, their relationships and root-set reachability. You can compare two heap snapshots to see where the memory growth has occurred.

  1. From the Targets menu, select Middleware, then click on a JVM or JVM Pool target.
  2. From the Java Virtual Machine or Java Virtual Machine Pool menu, select Heap Snapshots and Class Histograms.
  3. The list of available heaps is displayed. Highlight a help and click Detail. Click the Compare Heaps tab. The first heap in the list is selected for comparison and you are prompted to select the second heap.
  4. The two heaps are compared and a comparison table is displayed in the Diff Heaps page. The details of each heap with the following details are displayed:

    • Objects: The total number of objects reachable from the root.

    • KB: The total amount of memory reachable from the root.

    • Adj: The adjusted memory reachable from this root. This parameter is useful in tracking the memory leak hot-spots. It provides a better representation of the memory used by an object by ignoring backwards pointing references from child objects to their respective parent object.

    • Delta: The difference in the total and adjusted reachable memory.

  5. Click on the root-set with the most growth to diagnose the memory leak.
  6. Click the View Summary button to see a bottom up view of memory reachable by class of objects.
Viewing Heap Usage by Objects

Click the Usage tab to view the heap usage by objects. The following details are displayed:

  • Object Type: The type of object, Instance, Array, Klass, and so on.

  • Garbage: Indicates if this is garbage or reachable from root.

  • Objects: The total number of objects.

  • Total Memory: The total amount of memory reachable by root.

  • System: System details.

Click Compare with to compare the heap snapshot with another one. See Comparing Heap Snapshots.

Memory Leak Report

Click the Memory Leak Report tab to view the memory leak report.

The memory leak report shows the potential memory leak sources by finding frequent patterns in the heap graph. This tab shows a list of memory leak candidates which contain the most frequent patterns in a heap and could represent potential memory leak sources. Click Download Report to download the memory leak report in .txt format.

Anti-Pattern Report

Click the Anti-Pattern Report tab. The Anti-Pattern report is divided into different sections. Each section either shows the summary or one kind of anti-pattern issue. For example, the first section contains a summary of the most acute problems detected by JOverflow. The second section contains the total number of Java classes and Java objects. It also contains a histogram for top memory usage objects grouped by the Class. The third section shows the reference chains for high memory consumers. Each anti-pattern section calculates the overhead that shows the amount of memory that could be saved if the problem is eliminated.

Managing JFR Snapshots

Java Flight Recorder (JFR) provides a wealth of information on the inner workings of the JVM as well as on the Java program running in the JVM. You can use this information for profiling and for root cause analysis of problems. Furthermore, JFR can be enabled at all times, without causing performance overhead—even in heavily loaded, live production environments.

You can create JFR snapshots that include thread samples, which show where the program spends its time, as well as lock profiles and garbage collection details.

To create a JFR snapshot, follow these steps:

  1. From the Targets menu, select Middleware, then click on a Java Virtual Machine target.
  2. From the Java Virtual Machine menu, select the JFR Snapshots option.
  3. Click Create. In the Create JFR Snapshot window, enter a description, schedule the snapshot, and click Create. The newly created snapshot appears in the JFR Snapshots page. Fields in the table include:

    • Date: Date the JFR snapshot was created.

    • JVM Target: JVM Target associated with this JFR snapshot.

    • JFR Snapshot Description: Description of the JFR snapshot.

    • Host: Host containing the JFR snapshot.

    • Path: Path used to access the JFR snapshot

    • File Name: File containing the JFR snapshot.

Settings

To start, stop, and remove a JFR recording, highlight a JFR in the table, and click Settings.

The JFR Administration page provides the statistics of the recording: Status, Data End Time, Data Start Time, Destination Compressed, Destination File, Continuous Recording, Duration (sec), Maximum Size (KB), Maximum Age (sec), and Start Time.

View Reports

Click View Reports to view GC and Latency reports. Latency and GC reports are available when the data is detected.

Downloading a JFR Snapshot

Use JMC (Java Mission Control) to download and analyze the JFR snapshot. Select the JFR snapshot and click Download. You are prompted for the host credentials. Enter the credentials and click Download and specify the location on which the snapshot is to be saved.

Configuring a JVM

From the Targets menu, select Middleware, and then click on a Java Virtual Machine target. Select the Configure JVM Target option from the Java Virtual Machine menu. The Edit JVM Information page is displayed. You can change the JVM Pool, location of the Heap Dump Directory, and the Log Level. You can also modify the Bytecode Instrumentation (BCI). Click Save to save the changes.

Removing a JVM

You will see a warning message if you select the Remove Target option from the JVM menu. The message displays the name of the target being deleted. Click Yes to delete the JVM or No to return to the JVM Home page.

Adding a JVM to a Group

Select this option to add the JVM to one or more groups. A pop-up window appears with a list of groups on which you have Operator privileges. Select one or more groups and click Add to add the target to the group.

Managing Thread Snapshots

If a particular request is slow or hanging or if the entire application is slow, you can run the real-time transaction trace to view current Java application activity. You can look at the offending threads and their execution stack and analyze how much time a thread spent in waiting for DB wait or wait on a lock. Complex problems such as activity in one thread (or request) affecting the activity in the other thread or rest of the JVM can be found very quickly.

You can trace all active threads and generate a trace file that contains details such as resource usage, thread states, call stack information, and so on. During tracing, the state and stack of the target thread is sampled at set intervals for the desired duration. Follow these steps to trace active threads:

  1. From the Targets menu, select Middleware, then select a Java Virtual Machine target.
  2. Select the Thread Snapshots option from the Java Virtual Machine menu. The Thread Snapshots page appears.

    All the traces that have been loaded into the repository using the Trace Active Threads option are displayed here. For each thread, the Thread Snapshot ID, the date, JVM Name, Thread Name, Duration, and the number of samples taken during the trace is displayed. The Thread column indicates if all threads or only active threads have been traced.

  3. Click Create to take a thread snapshot of all active threads in the JVM. The Thread Snapshot of All Active Threads page appears where you can trace the active threads. See Tracing Active Threads for details.
  4. Select a thread and click the Details link to drill down to the Diagnostic Image Analysis page.
  5. Select a thread snapshot and click Import to upload a thread snapshot from your local machine. The Import Thread Snapshot dialog box is displayed. Click Browse and select the thread snapshot to be imported and click OK.

Tracing Active Threads

To trace active threads, follow these steps:

  1. Click Create in the Thread Snapshots page. The Thread Snapshot of All Active Threads page appears.
  2. Specify the following details:

    • Poll Interval (ms): The time interval between successive samples. The default value is 200 ms but can be changed.

    • Poll Duration (sec): The duration for which the thread snapshot should be taken.

    • Trace Thread Details: If this box is unchecked, only the last user call for the active thread will be stored. If the box is checked, all calls for the active thread will be stored, so you can view the call stack. Checking the box increases the overhead and space requirements

    • Try Changing Threads: If a thread stack changes during a sample (this can happen when a thread is using CPU), JVM Diagnostics will skip that thread for that sample. If you find missing samples, use this feature to retrace the changed stacks. This will retry (up to 5 times) threads with changing stacks. It will also make system calls to get the stack if possible.

    • Include Network Waits: Most JVMs have large number of idle threads waiting for network events. If you leave this check box unchecked, idle threads will not be included in the trace. Checking this box increases the overhead and space requirements.

    • Trace All Threads: Check this box if both idle and active threads will be included in the trace.

    • Allow Trace Interrupt: Allows you to interrupt the trace process.

  3. Click Take Thread Snapshot and click OK to generate the trace file. When the trace has been completed successfully, click here link to view the trace data in the Diagnostics Image Analysis page.

Analyzing Trace Diagnostic Images

A trace diagnostic image contains details such as resource usage, thread states, call stack information etc. The trace diagnostic image captures thread data at short intervals. If an application is hanging or is slow, you can analyze these threads and find out the application tier that causing the delay.

On the Diagnostic Image Analysis page, you can:

  • Click Description to view details of the thread snapshot being analyzed. The following Server State charts are displayed:

    • Active Threads by State: This chart shows the status of all threads in the JVM. The threads can be in different states like RMI, IO, NET, DB, CPU, and LOCK.

    • CPU Utilization by JVM: This chart shows the CPU utilization in the JVM.

    • Heap Utilization by JVM: This chart shows the heap utilization in the JVM.

  • You can filter the data that is displayed by specifying various criteria such as Method Name, JVM Name, Thread State, DBState, and so on. Check the Ignore Filters check box if you want to ignore the specified filters. The Active Threads by State, Top Requests, Top Methods, Top SQLs, Top DBWait Events, and Top Databases charts are displayed.

  • Click on the Threads tab to view the Thread State Transition, Metric By Active States, and Method data.

Viewing Heap Snapshots and Class Histograms

The JVM Diagnostics memory analysis feature allows you to not only find the objects responsible for the growth but also track their reachability from the root-set. With this feature, you can find the dangling reference responsible for memory leaks.To find a memory leak, you take snapshots of the JVM heap at different points in time. Between the snapshots, your JVM and Java applications continue running at full speed with zero overhead.

To view and analyze the heap usage, select Heap Snapshots and Class Histograms from the Java Virtual Machine Pool or Java Virtual Machine menu. The following regions are displayed:

  • Available Heap Snapshots: You can specify the Target Name and Target Type to filter the heap snapshots that are displayed. You can also specify the Heap ID in the Snap Name field to search for specific heap snapshots and display them. The following details is displayed:

    • Heap ID: The identification number for the heap snapshot.

    • Date: The date on which the heap snapshot was taken.

    • JVM Name: The server on which the JVM is running.

    • Size: The total size of the Java heap. An adequate heap size helps improve the performance of the application.

    • Used: The amount of heap that has already been used.

    • Used(%): The percentage of heap used.

    You can do the following:

    • Select a heap snapshot and click the Detail link to drill down to the Roots page. See Viewing Heap Usage by Roots.

    • Select a heap snapshot and click Load to load the heap snapshot to the repository.

    • Select a heap snapshot and click Reports to download heap reports to the local host. These reports must have been generated and loaded to the repository for the selected heap snapshot. You can download the Memory Leak Report and the Antipattern Report.

  • Available Class Histograms: The list of saved histograms with details such as date on which the snapshot was taken, Snap ID, Timestamp, JVM Name and Version, Description are displayed. The following options are available:

    • Details: Click this option to drill down to a detailed view of the heap.

    • Compare: Select two rows and click Compare. The Class Name, Instance Size (size of each snapshot), and Number of Instances (for each snapshot) are displayed.

JVM Offline Diagnostics

Diagnostic data for one or more JVM targets can be collected for a specific period and analyzed in an offline mode. This section describes the various options that are available to collect live JVM data and analyze it in offline mode. It contains the following sections:

Creating a Diagnostic Snapshot

You can create diagnostic snapshots for one or more JVM targets for a specified period. To create a diagnostic snapshot, specify the following:

  1. From the Targets menu, select Middleware.
  2. Select the Diagnostic Snapshots option from the Middleware Features menu.

    The Create Diagnostic Snapshot option is also available in the JVM Performance Diagnostics page. Navigate the Performance Diagnostics page for a JVM, specify the time range for which you want to create the collection and click Create Diagnostic Snapshot.

  3. Click Create in the Diagnostic Snapshots page. You can navigate to this page by clicking Offline Diagnostics on the Diagnostic Image Analysis page.
  4. Enter a name and description for the diagnostic snapshot.
  5. Specify the duration for the diagnostic snapshot.
  6. Click Add. Select one or more JVM targets for which the diagnostic data is to be collected.

    Note:

    The JVM targets that you select must belong to the same JVM Pool.

  7. Select the diagnostic types for the selected target and click OK. You will see a pop-up window that indicates that the diagnostic snapshot is being created. Click Close after the diagnostic snapshot has been created. You will return to the Diagnostic Snapshots page.

Using the Diagnostic Snapshots Page

You can collect diagnostic data for one or more JVM targets and analyze them in an offline mode. This page shows the list of diagnostic snapshots that have been created. You can specify search criteria to retrieve a specific snapshot. You can do the following:

  • Create: Click Create to create diagnostic snapshots for one or more JVMs. The Create Diagnostic Snapshot page is displayed.

  • Export: Select a file and click Export to export the diagnostic data to a file. Enter the location in which the file is to be stored. You can review and analyze the saved file in an offline mode on the same or a different host machine.

  • Import: Click Import to import an exported file with diagnostic data for a particular collection object. Specify the name of the file and upload the file from your system. You can analyze the exported file and view a summary of the diagnostic snapshot.

  • Analyze: Select a file and click Analyze. The Analyze Diagnostic Snapshot page is displayed.

  • Delete: Select a diagnostic snapshot from the list and click Delete. A confirmation message is displayed. Click OK to delete the diagnostic snapshot.

  • View: Select a file and click View. The View Diagnostic Snapshot page is displayed.

Analyzing a Diagnostic Snapshot

This page displays the summary details of the diagnostic snapshot and a summary of all the diagnostic types of the diagnostic snapshot. You can view the thread stack, thread states, CPU Utilization, Heap Utilization, Active Threads Graphs, and Garbage Collections.

To analyze a diagnostic snapshot, follow these steps:

  1. From the Targets menu, select Middleware.
  2. Select the Manage Diagnostic Snapshots option from the Middleware Features menu.
  3. In the Diagnostic Snapshots page, select a snapshot from the list and click Analyze.
  4. You can analyze details for each JVM for the specified time interval. Click More Details to view detailed diagnostics information for the JVM. The Diagnostic Image Analysis page is displayed.

Viewing a Diagnostic Snapshot

This page displays the summary of the targets, target types and the diagnostic information collected.

  1. From the Targets menu, select Middleware.
  2. Select the Manage Diagnostic Snapshots option from the Middleware Features menu.
  3. In the Diagnostic Snapshots page, select a snapshot from the list and click View.
  4. The summary details for the selected JVM target, target types, and the diagnostic information collected for the JVM is displayed.

Viewing JVM Diagnostics Threshold Violations

An event is a discrete occurrence detected by Enterprise Manager related to one or more managed entities at a particular point in time which may indicate normal or problematic behavior. Examples of events include: a database target going down, performance threshold violation, change in application configuration files, successful completion of job, or job failure.

JVM Diagnostics threshold violations are now integrated with the Enterprise Manager Event subsystem. When a threshold violation occurs, an Enterprise Manager event is generated. To view the event, follow these steps:

  1. From the Enterprise menu, select Monitoring, then select Incident Manager.
  2. In the View panel, click Events without Incidents. The JVM Diagnostics events are displayed if there are any outstanding JVMD threshold violations.
  3. Click on the link in the Target Name column of a JVM Diagnostics Event.

    The JVMD threshold violations will show up in the Incidents table of the JVM or JVM Pool Home page only if the events have been promoted to incidents. For more information on promoting events to incidents, see the Enterprise Manager Cloud Control Administrator's Guide.

Using Java Workload Explorer

Java Workload Explorer provides a detailed view of all performance statistics associated with the JVM and JVM Pool targets.

Accessing Java Workload Explorer

To use Java Workload Explorer:

  1. From the Targets menu, select Middleware, then select either a Java Virtual Machine target or a Java Virtual Machine Pool target.

    You can also access this page by selecting Middleware from the Targets menu and then selecting the Middleware Features menu. Select Java Workload Explorer.

  2. On the resulting page, click the Java Workload Explorer link at the top of the page.

Performance Analysis and Search Criteria

The Performance Analysis menu provides the following features:

Figure 17-4 Performance Analysis


performance analysis

  • Compare

    Compare available snapshots against current data or sets including data from multiple JVMs across domains. This enables you to compare current activity to a saved baseline snapshot. Use this to proactively spot deviations after new application deployments, upgrades, or configuration changes in the target JVM.

  • Targets

    Select the targets you want to analyze. Remove targets that no longer apply.

  • Sets

    Use this option to create, open, save, and manage sets against which to compare current data sets.

  • Java Workload Report

    Provides insight into the performance of the JVM in a selected time window. The report is available for a maximum of 10 targets with a duration of no more than one hour duration. Creating the report enables you to analyze the data with data reported at different points in time.

    The following tables are displayed in the Java Workload Report:

    Summary tables for each target include:

    • JVM Summary

    • Diagnostic Findings

    • Threshold Violations

    • JVM Statistics

    • OS Statistics

    • GC Statistics

    Multiple tables aggregated by all targets in the context and sorted by important metrics include:

    • Requests Statistics

    • Request Instances Statistics

    • Session Statistics

    • User Statistics

    • Application Statistics

    • Thread Statistics

    • Method Statistics

    • Class Statistics

    • Packages Statistics

    • Databases Statistics

    • SQLs Statistics

    • Database Events Statistics

    • Database Schema Statistics

    • Database Modules Statistics

    • Database Actions Statistics

    • Other External Resources Statistics

    • Locks Statistics

    • Files Statistics

    • Supplemental Information

      Contains JVM startup parameters, full SQLs, and Stacks

Search Criteria

The Search Criteria provided throughout the page enables you to fine tune your search and minimize the reported data.

Figure 17-5 Search Criteria


Add to Search

By default, the following keywords are used: ECID Duration (ms), SQL Duration (ms), and State. Using State enables you to select the Thread State in which you have interest. During any search, you can elect to ignore the field.

Using the Add Field menu, you can select fields for any of the following: request, user session, database, internal resource, code, and threads.

Graph Highlights

The graph at the top of the page provides a visual representation of the workload. Use this graph to quickly narrow down the time selection to the interval of interest. By default, the graph provides statistics on the active threads: RMI Wait, I/O Wait, Network Wait, DB Wait, CPU, and Lock. The data in the graph is available in table format.

Using the Graph Metric menu, you can narrow the graph to report on Memory Utilization, CPU Utilization, GC (garbage collection) Overhead, and Response and Load.

The Graph Height menu enables you to adjust the graph to display more details on the statistic.

The Graph Resolution menu enables you to see more spikes in the chart by increasing the number of data-points on the time axis (x-axis).

Diagnostics

The statistical data associated with the JVM or JVM Pool is available in the form of tabs. A diagnostic tab corresponds to a user intention based on a region or a set of related regions. A tab can have associated subtabs.

The majority of the tabs have an Action menu and a View menu. The options on the Action menu often replicate the options available on other parts of the screen, most notably Add to Search and Add to Set. Additional menu options are:

  • Add to Search: It adds the element to the Search Criteria.

    Figure 17-6 Add to Search


    add to search

  • Add to Set: It adds the element to the Set Criteria.

    Figure 17-7 Add to set


    add to set

  • Log Viewer (Displays Log Messages).

  • View Call Tree (Displays the methods and the percentage of time for the call to execute to method).

  • View Thread Transition (Displays the graphical view of how threads change over time).

  • Session Diagnostics (This is a RUEI (Real User Experience Insight) based analysis and will be enabled only if the JVM target has been enabled on a RUEI system).

The tabs are:

  • Overview:

    Figure 17-8 Overview Tab


    overview

    Statistics include: OSR, Context Switch (per sec), Host Memory (%), Swap Space (%), Open File Descriptors (%), Max Heap Size (MB), Min Heap Size (MB), Container Type, Container Name, and Version

    Actions available are: Add to Search, Log Viewer, and Diagnostic Findings.

  • Requests, ECIDs, Sessions, and Users.

    • Requests

      Figure 17-9 Request Tab


      request tab

      Statistics include: Duration (ms), Max. Duration (ms), JVM CPU (sec), Allocation (MB), Count, JVM Time (sec), and Thread State.

      Sample Request metrics (e.g. count, Allocation, Duration) are calculated based on data collected from specific instances that are caught while taking a sample of the stuck. The non-sample metrics are calculated based on all the instances that were executed since the previous sampling.

      For example:

      A Request average execution time varies from 50 to 1500ms, while the average execution time is 100ms. The request is executed 1000 times per second.

      Sampling catches mainly the slow executions. The sampled count will be much smaller than 1000/s and the average execution time and other metrics will reflect the behavior of the caught slow executions. With the new feature introduced in 13.3, all the Request executions are counted and measured. In the example above, the count will show 100/s and the average will be 100ms.

      Note:

      • The Thread state and JVM Time metrics are available only as sampled data.

      • Request that are not sampled at all (not even one instance is seen the thread stuck when it is sampled by JVMD) will not show at all for that 2 second time period.

      Actions available are: Add to Search, View Call Tree, Add to Set, and View Thread Transition.

    • Applications

      Statistics include: JVM Time (sec), Thread State, and Application Name.

      Actions available are: Add to Search, Add to Set.

    • Users

      Statistics include JVM Time (sec), Thread State, and User.

      Actions available are: Add to Search, Add to Set, and Log Viewer,

    • Sessions

      Statistics include: Minor GC Time (ms), Minor GC Count, Major GC Time (ms), Major GC Count, JVM Time (sec), Thread State, Number of Requests, User, and Session ID.

      Actions available are: Add to Search, View Call Tree, Add to Set, and View Thread Transition.

    • Request Instances

      Statistics include: Minor GC Time (ms), Minor GC Count, Major GC Time (ms), Major GC Count, GC Overhead (ms), Allocation (MB), JVM CPU (sec), Duration (ms), JVM Time (sec), Thread State

      Actions available are: Add to Search, View Cal Tree, Add to Set, Log Viewer, and View Thread Transition, Session Diagnostics.

  • External Resources

    Figure 17-10 External Resources Tab


    external resources

    • External Resources Overview

      Network Wait and Database (JVM Time (% of Total), JVM Time (% of Internal), and JVM Time (sec))

    • Databases

      Statistics include: Max Duration (ms), Avg. Duration (ms), Count, JVM Time (sec), and More Information, Database

      Actions available are: Add to Search, View Call Tree, Add to Set, and Database Drill Down.

    • SQL Queries

      Statistics include: Max Duration (ms), Avg. Duration (ms), JVM Time (sec), Database, SQL ID, and SQL Statement

      Actions available are: Add to Search, View Call Tree, Add to Set, and SQL Details

    • Database Events

      Statistics include: Max Duration (ms), Avg. Duration (ms), Count, JVM Time (sec), Database, and Database Event

      Actions available are: Add to Search, View Call Tree, Add to Set

    • Database Schemas

      Statistics include: Max Duration (ms), Avg. Duration (ms), Count, JVM Time (sec), Database Schemas

      Actions available are: Add to Search, View Call Tree, Add to Set

    • Database Modules

      Statistics include: Max Duration (ms), Avg. Duration (ms), Count, JVM Time (sec), Database Module

      Actions available are: Add to Search, View Call Tree, Add to Set

    • Database Actions

      Statistics include: Max Duration (ms), Avg. Duration (ms), Count, JVM Time (sec), Database Action

      Actions available are: Add to Search, View Call Tree, Add to Set

    • Other External Resources

      Statistics include: JVM Time (sec), Protocol, Request

      Actions available are: Add to Search, Add to Set

  • Internal Resources

    Figure 17-11 Internal Resources Tab


    Internal resources

    • Internal Resources Overview

      • CPU - JVM Time (% of Total), JVM Time (% of Internal), and JVM Time (sec)

      • Lock - JVM Time (% of Total, JVM Time (% of Internal), JVM Time (sec)

      • I/O File - JVM Time (% of Total), JVM Time (% of Internal), and JVM Time (sec)

    • Locks

      Statistics include: Held Locks (JVM Time (sec), Avg. Duration (ms), Max Duration (ms) and Waiting Locks (Thread Trend, JVM Time (sec), Avg. Duration (ms), and Max Duration (ms)

      Actions available are: Add to Search, View Details, Add to Set

    • Files

      Statistics include: I/O file and JVM Time (sec)

      Actions available are: Add to Search, Add to Set

  • Code

    Figure 17-12 Code Tab


    Code

    Statistics include: % of Total, JVM Time (sec), and Package

    • Methods

      Actions available are: Add to Search, View Call Tree, Add to Set

    • Classes

      Actions available are: Add to Search, Add to Set

    • Packages

      Actions available are: Add to Search, Add to Set

  • Threads

    Figure 17-13 Threads Tab


    threads

    Statistics include: Write Characters, Read Characters, Wait Count, Waited Time (sec), Blocked Count, Blocked Time (ms), Hogger (%), Stuck (%), Allocation (MB per sec), JVM CPU (per minute)

    Actions available are: Add to Search, View Call Tree, Add to Set, Log Viewer, View Thread Transition, Sample Analysis

Managing and Troubleshooting JVMD (Globally)

If you find that a particular JVM Diagnostics Engine is exhibiting problems, the Manage and Troubleshoot JVMD functionality provides the statistics and diagnostic aids to help resolve the issue.

To access the Manage and Troubleshoot JVMD page:

  1. From the Setup menu select Middleware Management, then select Engines and Agents .
  2. In the RUEI/JVMD Engines section, highlight the JVM Diagnostic Engine of interest and click Troubleshoot.

    Statistics include:

    • Repository Statistics

      The Tablespace Growth Rate chart provides the Total Space and Used Space used by the repository over specific time intervals. The related repository tables are listed. To view the statistics of a particular repository table, highlight the table name and click Details.

      Click the Trend button to view the used and allocated data for each date. This data is based on the statistics collected by the DBMS_SPACE.OBJECT_GROWTH_TREND function and enables you to see trends in the usage of the space.

      Click Export to retrieve a table listing all the tables and their associated statistics, for example, Table Allocated Space (MB), Index Allocated Space (MB), Number of Rows, and Last Analyzed Time. Note: Before clicking Export, show all the columns (on the View menu, select Columns, then select Show All). This provides a better view of the columns in the table.

      The data provided in the JVMD Operations Statistics region enables you, as the JVMD Administrator, to monitor your own applications.

    • JVM Target Summary

      This page provides data about the JVM Agent.

      Summary section lists agent statistics such as the number of targets that are down, and the number of unassociated targets. You can manage JVMD Agents located on WebLogic Server Domains, start and stop monitoring of a JVM target, and export data.

    • Engine Summary

      This page provides statistics regarding the JVM engine. When you highlight the engine, the associated attributes display in the Engine Attributes table. The engine summary includes the following types of attributes: Performance, Diagnostics, and Configuration.

      Also, if there are any load balancers configured, the JVMD Load Balancer table provides additional information.

    Troubleshooting diagnostic aids include:

    • View JVMD Health Jobs

      This link directly navigates to the Job system page and by default shows all the JVMD health jobs.

      The JVMDHealthReportJob job is automatically invoked every three hours. This job collects statistics for that three hour time period. Select the job for the time period of interest and click View Results. This is an historical view of the health of the JVMD. Name of the report is JVMD_HEATH_REPORT_AUTO.

    • SR Assistance

      In the event that there are issues with the JVMD, click the SR Assistance button for an explanation regarding common JVMD issues. This page also lists the statistics you need to have available before filing a Service Request.

    • Generate Report

      Provides the same information as is available in the Manage and Troubleshoot JVMD tabs, that is, it shows the trends of the various JVMD components. Click Save to File to save the information to an .html file that you can easily access at a later time.

Managing and Troubleshooting JVMD (Specific Agent)

Should you find that a particular JVM or JVM Pool is sluggish or is posing problems, the Manage and Troubleshoot JVMD functionality provides the statistics and diagnostic aids to help resolve the issue.

To access the Manage and Troubleshoot JVMD page:

  1. From the Targets menu, select Middleware, then select a Java Virtual Machine or Java Virtual Machine Pool.
  2. On the resulting page, select Manage and Troubleshoot JVMD from the Java Virtual Machine or Java Virtual Machine Pool menu.

    JVM Target Summary Tab

    Statistics include:

    • Status and Connection

      Data includes the engine host and availability, as well as JVMD Agent status, Monitoring status, and Bytecode Instrumentation (BCI) status.

    • Target Attributes related to target. Attributes include: Performance, Diagnostics, and Configuration attributes.

      Click the MBean Browser button to view JVMD Agent MBean data and it's live call to the JVMD Agent.

      This data can be exported which is very helpful in diagnosing the JVMD Agent related issues.

    • Performance and Diagnostics (Poll Interval (ms), Response Time (ms), Average Stack Depth (count), Number of Active Threads)

    Troubleshooting diagnostic aids include:

    • Java Virtual Machine menu

      Provides links to performance diagnostics, thread snapshots, and configuration options.

    • Java Workload Explorer

      Provides a detailed view of all performance statistics associated with the JVM or JVM Pool.

    • Live Thread Analysis

      Shows the real-time data for all the JVMs in the selected pool or the real time data for the selected JVM.

    • SR Assistance

      In the event that there are issues with the JVMD, click the SR Assistance button for explanation regarding common JVMD issues. This page also lists the statistics you need to have available before filing a Service Request.

    • Generate Report

      Provides the same information as is available in the Manage and Troubleshoot JVMD tabs. Click Save to File to save the information to an .html file that you can easily access at a later time. Purpose of job is that it shows the trends of the various JVMD components.

    Manage Association Tab

    Target Association lists the Enterprise Manager targets with which this JVM is associated. You can associate and disassociate targets, and export the information to a spreadsheet.

Enable or Disable Monitoring of JVM Targets using EMCLI

You can also enable or disable the monitoring of JVM target using EMCLI.

Run the following command to deploy JVMD targets:

emcli deploy_jvmd -domain_name="/Farm03_base_domain/base_domain" -enableMonit="false"