4 Using WLDF with Java Flight Recorder
This chapter also explains common usage scenarios that show how this integration can provide for a comprehensive performance analysis and diagnostic foundation for production systems based on WebLogic Server.
- About Java Flight Recorder
Java Flight Recorder is a performance monitoring and profiling tool that records diagnostic information on a continuous basis. The Java Flight Recorder is available even when there is a catastrophic failure such as a system crash. - Using Java Flight Recorder with Oracle HotSpot
Java Flight Recorder is available with Oracle Hotspot. If WebLogic Server is configured with Oracle HotSpot, Java Flight Recorder is disabled by default. Enable the Java Flight Recorder to capture the WLDF diagnostic data. - Key Features of WLDF Integration with Java Flight Recorder
WLDF integration with Java Flight Recorder provides several useful features, including having WebLogic Server events captured in the flight recording, the ability to throttle the volume of data captured, tools for downloading diagnostic image captures, and more. - Java Flight Recorder Use Cases
Java Flight Recorder helps to resolve important diagnostic issues such as diagnosing critical failure, and examining and reporting runtime data. When a critical failure occurs, the data captured by Java Flight Recorder is useful for failure analysis. Likewise, capturing data at specific time and at runtime help to diagnose data after and before a particular event. - Obtaining the Flight Recording File
The diagnostic image capture is a single Java Flight Recorder (JFR) file that contains individual images produced by different server subsystems. The JFR file is included in the diagnostic image asFlightRecording.jfr.
About Java Flight Recorder
Java Flight Recorder is a performance monitoring and profiling tool that records diagnostic information on a continuous basis. The Java Flight Recorder is available even when there is a catastrophic failure such as a system crash.
Java Flight Recorder is available in Oracle HotSpot. When WebLogic Server is configured with HotSpot, Java Flight Recorder is not enabled by default. See Using Java Flight Recorder with Oracle HotSpot for information about how to enable Java Flight Recorder with WebLogic Server.
Note:
For the most current information about configurations supported in this release of WebLogic Server, see Oracle Fusion Middleware Supported System Configurations on the Oracle Technology Network.
Java Flight Recorder maintains a buffer of diagnostics and profiling data, called a flight recording or a JFR file, that you can access whenever you need it. The flight recording functions in a manner similar to an aircraft "black box" in which new data is continuously added and older data is stripped out, as shown in Figure 4-1.
Figure 4-1 Circular Flight Recording Buffer
Description of "Figure 4-1 Circular Flight Recording Buffer"
The data contained in the JFR file includes events from the JVM and from any other event producer, such as WebLogic Server and Oracle Dynamic Monitoring System (DMS). The JFR file can be analyzed at any time, using Java Mission Control, to examine the details of system execution flow that occurred leading up to an event.
The amount of additional processing overhead that results when Java Flight Recorder is enabled, and also configure WLDF to generate WebLogic Server diagnostics to be captured by Java Flight Recorder, is minimal. This makes it ideal to be used on a full time basis, especially in production environments where it adds the greatest value.
Java Flight Recorder provides the following key benefits:
-
Designed to run continuously — When Java Flight Recorder is configured to run full-time, with both JVM and WLDF events captured in the flight recording, diagnostic data is always available at the time an event occurs, including a system crash. This ensures that a record of diagnostic data leading up to the event is available, allowing you to diagnose the event without having to recreate it.
-
Comprehensive data — Java Flight Recorder combines data generated by tools such as the Runtime Analyzer and the Latency Analysis Tool and presents it in one place.
-
Integration with event providers — HotSpot includes a set of APIs that allow Java Flight Recorder to monitor additional system components, including WebLogic Server, Oracle Dynamic Monitoring System (DMS), and other Oracle products.
For more information about Java Flight Recorder, see Java Flight Recorder Runtime Guide at the following location:
Parent topic: Using WLDF with Java Flight Recorder
Using Java Flight Recorder with Oracle HotSpot
Java Flight Recorder is available with Oracle Hotspot. If WebLogic Server is configured with Oracle HotSpot, Java Flight Recorder is disabled by default. Enable the Java Flight Recorder to capture the WLDF diagnostic data.
To enable Java Flight Recorder, you must specify the following JVM options in the WebLogic Server instance in which the JVM runs:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
Note:
The sequence in which you specify JVM options to Hotspot is very important. The options are processed from left to right, and option values are overwritten if there are duplicates. Therefore, note the following:
-
HotSpot does not recognize the
FlightRecorderoption unless it is preceded by theUnlockCommercialFeaturesoption. -
If you specify only the
FlightRecorderoption, or you specifyFlightRecorderbefore specifyingUnlockCommercialFeatures, the HotSpot JVM does not start.
Parent topic: Using WLDF with Java Flight Recorder
Key Features of WLDF Integration with Java Flight Recorder
WLDF integration with Java Flight Recorder provides several useful features, including having WebLogic Server events captured in the flight recording, the ability to throttle the volume of data captured, tools for downloading diagnostic image captures, and more.
The key features provided by WLDF to leverage integration with Java Flight Recorder include the following:
-
WLDF diagnostic data captured in a flight recording
WLDF can be configured to generate diagnostic data about WebLogic Server events that is captured in the flight recording. Captured events include those from components such as: web applications; EJBs; JDBC, JTA, and JMS resources; resource adapters; and WebLogic web services.
-
WLDF diagnostic volume control
The ability to generate WebLogic Server event data for the Flight Recording is controlled by the WLDF diagnostic volume configuration. This control also determines the amount of WebLogic Server event data that is captured by Java Flight Recorder, and can be adjusted to include more, or less, data for each WebLogic Server event that is generated.
Note:
-
By default, the WLDF diagnostic volume is set to
Low. -
The WLDF diagnostic volume setting does not affect explicitly configured diagnostic modules or the built-in diagnostic modules.
-
-
Automatic throttling of generated events under load
As processing load rises on a given WebLogic Server instance, WLDF automatically begins throttling the number of incoming WebLogic Server requests that are selected for event generation and recording into the JFR file. The degree of throttling is adjusted continuously as system load rises and falls.
Throttling provides three key benefits:
-
The overhead of capturing events generated by WLDF for Java Flight Recorder remains minimized, which is especially important when systems are under load.
-
The time interval encompassed in the flight recording buffer is maximized, giving you a better historical record of data.
-
Throttling has the effect of sampling incoming WebLogic Server requests, maintaining high performance while still providing an accurate overall view of system activity under load.
Note:
Throttling affects only the Flight Recording data that is captured by WLDF. It does not affect data captured by other event producers, such as the JVM.
-
-
WLDF diagnostic image capture support for JFR files
WLDF diagnostic image capture automatically includes the JFR file, if one has been generated by Java Flight Recorder. The JFR file includes data generated by all active event producers, including WebLogic Server. An image captured using the Policies and Actions component may contain the JFR file, if available.
-
WLST commands for downloading the contents of diagnostic image captures
WLST includes a set of commands for downloading the contents of diagnostic image captures, described in WLST Online Commands for Downloading Diagnostics Image Captures. Although these commands are generally useful for listing, copying, and downloading all entries contained in the diagnostic image capture, they can also be used for obtaining the JFR file, if available. Once obtained from the diagnostic image capture, the JFR file can be viewed in Java Mission Control.
Parent topic: Using WLDF with Java Flight Recorder
Java Flight Recorder Use Cases
Java Flight Recorder helps to resolve important diagnostic issues such as diagnosing critical failure, and examining and reporting runtime data. When a critical failure occurs, the data captured by Java Flight Recorder is useful for failure analysis. Likewise, capturing data at specific time and at runtime help to diagnose data after and before a particular event.
This section summarizes the three common business cases of using the Java Flight Recorder to resolve diagnostic issues:
For more information about scenarios using Java Flight Recorder, see also About Java Flight Recorder in Java Flight Recorder Runtime Guide, available at the following URL:
- Diagnosing a Critical Failure — The "Black Box"
- Profiling During Performance Testing or in Production
- Real-Time Application Diagnostics and Reporting
Parent topic: Using WLDF with Java Flight Recorder
Diagnosing a Critical Failure — The "Black Box"
When a "catastrophic" failure occurs, the content of the Java Flight Recorder buffer can be made available for post-failure analysis in a manner analogous to the use of an aircraft's black box. Examples of such failures include a JVM crash or an out-of-memory error (OOME) resulting in an application terminating.
When these situations arise, the flight recording contains the following information, which can be helpful in determining the cause of the failure:
-
JVM core dump, including metadata about the Java Flight Recorder configuration at the time of the crash. Furthermore, depending on the disk storage parameters that are set, the Java Flight Recorder data buffer might contain a certain amount of data.
-
WebLogic Server events, captured by WLDF, that preceded the failure.
Java Flight Recorder uses a combination of memory and disk to store its buffer. The most recent data is stored in memory and is flushed out to disk as it "ages". In this way, the on-disk data can be available even after a power failure or similar catastrophic event; only the most recent data will be unavailable (for example, the data that had not yet been flushed to disk). The text dump file will contain metadata about the Java Flight Recorder configuration at the time of the crash, including the path to the data buffer file when applicable.
Parent topic: Java Flight Recorder Use Cases
Profiling During Performance Testing or in Production
Profiling involves capturing data beginning at a specific point in time so that, later, you can analyze the events that were generated after that point. In contrast to real-time diagnostics reporting, described in the following section, profiling involves analyzing the diagnostic data generated after a particular event occurs, as opposed to the data that precedes it.
Profiling with Java Flight Recorder optimizes the ability to perform deep analysis of lock contention and causes of latency.
Parent topic: Java Flight Recorder Use Cases
Real-Time Application Diagnostics and Reporting
It is particularly useful to examine diagnostic data generated during run time when a particular event occurs for the purposes of understanding the system activity that preceded the event; for example, system activity occurring moments before a serious error message is generated. By using the diagnostic capabilities available in WLDF in conjunction with Java Flight Recorder, you can capture a large amount of system-wide diagnostic data the moment a problem occurs. You can then leverage the capabilities of Java Mission Control to quickly correlate that event with other system activity and process execution data within the "snapshot in time" that the JFR file provides, enabling you to quickly isolate likely causes of the problem.
One WLDF feature that is particularly useful in conjunction with Java Flight Recorder is the image action. An image action generates a diagnostic image capture in response to the triggering of a policy that is configured in a diagnostic system module. The policy monitors the server environment for one or more specific conditions, and when those conditions occur, the policy can automatically executes an image action. When Flight Recorder is enabled, the diagnostic image capture automatically includes the JFR file. The JFR file can then be extracted from the diagnostic image capture and examined immediately in Java Mission Control or stored for later analysis. An image action, used when WLDF data is captured by Java Flight Recorder, is particularly well suited for real-time diagnosis of intermittent problems.
Image action is part of the Policies and Actions system in WLDF. To set up an image action, you create one or more individual policies. A policy includes a Java EL expression to specify the event for the policy to detect. For example, the following log policy expression detects the server log message with severity level Critical and ID BEA-149618:
log.severityString == 'Critical' && log.messageId == 'BEA-149618'
Policies can monitor any of the following:
-
Runtime MBean instances in the local runtime MBean server
A scheduled policy can execute an image action if runtime MBean attributes detect a performance issue, such as high memory utilization rates or problems with open socket connections to the server.
-
Messages published to the server log
A log policy can execute an image action if a specific message, severity level, or string is issued.
-
Event generated by the WLDF Instrumentation component
An event policy can execute an image action if an instrumentation service generates a particular event.
See the following topics:
The following section explains how to obtain the JFR file from the diagnostic image capture:
Parent topic: Java Flight Recorder Use Cases
Obtaining the Flight Recording File
The diagnostic image capture is a single Java Flight Recorder (JFR) file that contains individual images produced by different server subsystems. The JFR file is included in the diagnostic image as FlightRecording.jfr.
A diagnostic image capture can be generated on-demand — for example, from the WebLogic Remote Console, Fusion Middleware Control, WLST, or a JMX application — or it can be generated as the result of an image action.
To view the contents of the JFR file, you first need to extract it from the diagnostic image capture as described in Configuring and Capturing Diagnostic Images. Once you have extracted the JFR file, you can view its contents in Java Mission Control.
For an example WLST script that retrieves the JFR file from a diagnostic image file and saves it to a local directory, see Example: Retrieving a JFR File from a Diagnostic Image Capture.
Parent topic: Using WLDF with Java Flight Recorder