13 Diagnosing Problems

13.1.1 About Incidents and Problems

To facilitate diagnosis and resolution of critical errors, the Diagnostic Framework introduces two concepts for Oracle Fusion Middleware: problems and incidents.

A problem is a critical error. Critical errors manifest as internal errors or other severe errors. Problems are tracked in the ADR. Each problem has a problem key, which is a text string that describes the problem. It includes an error code (in the format XXX-nnnnn) and in some cases, other error-specific values.

An incident is a single occurrence of a problem. When a problem (critical error) occurs multiple times, an incident is created for each occurrence. Incidents are timestamped and tracked in the ADR. Each incident is identified by a numeric incident ID, which is unique within the ADR home. When an incident occurs, the Diagnostic Framework:

• Gathers first-failure diagnostic data about the incident in the form of dump files (incident dumps).

• Stores the incident dumps in an ADR subdirectory created for that incident.

• Registers the incidents dumps with the incident in ADR.

13.1.2 Diagnostic Framework Components

The following topics describe the key components of the Diagnostic Framework:

• Automatic Diagnostic Repository

• Diagnostic Dumps

• Management MBeans

• WLST Commands for Diagnostic Framework

• ADRCI Command-Line Utility

DOMAIN_HOME/SERVERS/server_name/adr

13.1.2.1 Automatic Diagnostic Repository

The Automatic Diagnostic Repository (ADR) is a file-based hierarchical repository for Oracle Fusion Middleware diagnostic data, such as traces and dumps. The Oracle Fusion Middleware components store all incident data in the ADR. Each Oracle WebLogic Server stores diagnostic data in subdirectories of its own home directory within the ADR. For example, each Managed Server and Administration Server has an ADR home directory.

The ADR root directory is known as ADR base. By default, the ADR base is located in the following directory:

DOMAIN_HOME/servers/server_name/adr

Within ADR base, there can be multiple ADR homes, where each ADR home is the root directory for all incident data for a particular instance of Oracle WebLogic Server. The following path shows the location of the ADR home:

ADR_BASE/diag/ofm/domain_name/server_name

Figure 13-1 illustrates the directory hierarchy of the ADR home for an Oracle WebLogic Server instance.

Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware

Description of "Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware"

The subdirectories in the ADR home contain the following information:

• alert: The XML-formatted alert log.

• incident: A directory that can contain multiple subdirectories, where each subdirectory is named for a particular incident. The subdirectories are named incdir_n, with n representing the number of the incident. Each subdirectory contains information and diagnostic dumps pertaining only to that incident.

• (others): Other subdirectories of ADR home, which store incident packages and other information.

Note:

ADR uses the domain name as the Product ID and the server name as the Instance ID when it packages an incident. However, if either name is more than 30 characters, ADR truncates the name. In addition, dollar sign ($) and space characters are replaced with underscores.

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

13.3.1 Configuring Diagnostic Framework Settings

You can configure the following settings:

• Enabling or disabling the detection of incidents through the log files

• Enabling or disabling flood control and setting parameters for flood control

You configure these settings by using the Diagnostic Framework MBean DiagnosticConfig. The following shows the MBean's ObjectName:

oracle.dfw:type=oracle.dfw.jmx.DiagnosticsConfigMBean,name=DiagnosticsConfig 

Table 13-1 shows the attributes for the DiagnosticConfig MBean and a description of each parameter.

The following example shows how to configure these settings using the Fusion Middleware Control System MBean Browser:

1. From the target navigation pane, expand the farm, then WebLogic Domain.

2. Select the domain.

3. From the WebLogic Domain menu, choose System MBean Browser.

   The System MBean Browser page is displayed.

4. Expand Application Defined Beans, then oracle.dfw, then domain.domain_name, then dfw.jmx.DiagnosticsConfigMBean.

5. Select one of the DiagnosticConfig entries. There is one DiagnosticConfig entry for each server.

6. In the Application Defined MBean pane, expand Show MBean Information to see the server name.

   The following shows the System MBean Browser page:

   
   Description of the illustration dfw_config.gif
   
   

7. To change the values for the attributes listed in Table 13-1, enter or select the value in the Value field.

8. Click Apply.

13.3.2 Configuring Custom Diagnostic Rules

You can configure custom diagnostic rules that apply to a domain, a server, or an application in a domain or server.

You create the custom diagnostic rules by creating an .xml file with a particular format, which is shown in the example later in this section. You must save the file to one of the following locations:

• For rules that apply to the entire domain:

  DOMAIN_HOME/config/fmwconfig/dfw
  

• For rules that apply to a particular server:

  DOMAIN_HOME/config/fmwconfig/servers/server_name/dfw
  

The file name must use the following format:

name.xml
appname#name.xml

In the format, appname is the name of the application to which the rule applies. The appname must be the exact name of the deployed application. name is the name of the rule you specify. If you do not specify appname, the rules will apply to the entire server. For example, the following rule applies to the application myApp:

myApp#custom_rule.xml

The custom diagnostic rules file can contain the following types of elements to define the rule:

• Log detection conditions, which is optional

  You can define a set of conditions, in the logDetectionConditions element, to check for in the diagnostic logs applicable to the server or to the specified application against which that the rules are registered. When a log message matching the condition is detected, an incident is created, capturing diagnostics that will help identify the problem. By default, all INCIDENT_ERROR messages are detected and an incident created for them. In addition, specific components may have configured rules to detect specific messages.

  The following example shows a fragment of a custom diagnostic rules file that defines four log detection conditions. If one or more of the conditions are true, an incident is created.

  <?xml version="1.0" encoding="UTF-8"?>
  <diagnosticRules xmlns="http://www.oracle.com/DFW/DiagnosticsFrameworkRules/" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance">
    <logDetectionConditions>
        <condition messageSeverity="INCIDENT_ERROR"/> 
        <condition messageSeverity="ERROR" component="jrfServer_admin"/> 
        <condition messageSeverity="ERROR" module="test.servletA"/> 
        <condition messageId="FMW-40300"/> 
      </logDetectionConditions>
  

  See Table 13-2 for a description of the conditions you can use.

• Processing rules

  You can define processing rules that are evaluated when either the server or application rules are involved in incident creation. For example, if the application MyApp is involved in incident creation, any rules associated with the MyApp application are evaluated. In all cases, server-wide rules are evaluated regardless of the application.

  Processing rules consist of two parts:

  • Default actions, which are optional. If they are present, they are always executed during incident creation. The actions are a list of diagnostic dumps to execute, along with optional arguments.

    The following shows an example set of default actions:

    <defaultActions>
          <dumpAction name="odl.logs">
            <argument name="timestamp" value="INCIDENT_TIME" valueType="fact"/>
          </dumpAction>
          <dumpAction name="dms.metrics"/>
        </defaultActions>
    

    See Table 13-3 for a description of the optional arguments that you can use.

  • Condition-based actions, which are executed only if the condition evaluates to true. Each <rule> element consists of a name attribute, along with a child <ruleCondition> element and a child <ruleActions> element. The <ruleActions> element contains one or more dumpAction elements. See Table 13-4 for a list of the <ruleCondition> element attributes.

    If multiple <condition> elements are specified in a single <rule> element, the dumpAction is executed only if all conditions evaluate to true.

    The following shows an example of a condition-based action rule. If the MESSAGE_ID is DFW-99997, the condition evaluates to true and the jvm.classhistogram dump is executed.

    <processingRules>
          <rule name="OOME">
            <ruleCondition>
              <condition name="MESSAGE_ID" value="DFW-99997"/>
            </ruleCondition>
            <ruleActions>
              <dumpAction name="jvm.classhistogram"/>
            </ruleActions>
          </rule>
        </processingRules>
    

Table 13-2 describes the attributes you can use to create the log detection conditions:

See Table 12-1 for a description of the ODL log file fields.

Table 13-3 describes the optional arguments that you can use for the <defaultActions> element.

Table 13-4 shows the <ruleCondition> element attributes.

To create and load a custom diagnostic rule:

1. Create a file that contains the custom rules.

   The following shows a sample custom rules file:

   <?xml version="1.0" encoding="UTF-8"?>
   <diagnosticRules xmlns="http://www.oracle.com/DFW/DiagnosticsFrameworkRules/" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance">
    
        
       <logDetectionConditions>
         <condition messageSeverity="INCIDENT_ERROR"/> 
              // detect all message logged at level INCIDENT_ERROR
         <condition messageSeverity="ERROR" component="jrfServer_admin"/> 
             // detect all "jrfServer_admin" component messages logged at level ERROR
         <condition messageSeverity="ERROR" module="test.servletA"/> 
             // detect all "test.servlet" module messages logged at level ERROR
         <condition messageId="FMW-40300"/> 
             // detect message "FMW-40300"
       </logDetectionConditions>
       
       <defaultActions>
         <dumpAction name="odl.logs">
           <argument name="timestamp" value="INCIDENT_TIME" valueType="fact"/>
         </dumpAction>
         <dumpAction name="dms.metrics"/>
       </defaultActions>
    
       <processingRules>
         <rule name="OOME">
           <ruleCondition>
             <condition name="MESSAGE_ID" value="DFW-99997"/>
           </ruleCondition>
           <ruleActions>
             <dumpAction name="jvm.classhistogram"/>
           </ruleActions>
         </rule>
       </processingRules>
    
   </diagnosticRules>
   

2. Save the file, naming it with the extension .xml. If the rule applies to an application, precede the file name with app_name#. Save the file to one of the following locations:

   DOMAIN_HOME/config/fmwconfig/dfw
   DOMAIN_HOME/config/fmwconfig/servers/server_name/dfw
   

3. Load the rules, using the WLST command reloadCustomRules. The following example loads the rule customrules.xml, which applies to the application myApp:

   reloadCustomRules(name='myApp#customrules.xml')
   

   You can reload all the rules in the domain or all the rules that pertain to a particular server. The following example reloads all the rules for the server soa_server1:

   reloadCustomRules(server='soaserver1')
   

   For more information about the reloadCustomRules command, see "reloadCustomRules" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

13.3.3 Configuring Problem Suppression

In certain situations, you may want to suppress the creation of incidents based on a particular problem key. For example, in a development environment, when you are developing a servlet, you may generate high number of uncaught exceptions as you refine the code. This results in the creation of unnecessary incidents.

The Diagnostic Framework allows you to configure problem suppression filters so that problems that match the filter criteria do not result in the creation of an incident.

When you configure a problem suppression filter, you use a regular expression that represents a pattern that you want to match. The regular expression is matched using the java.util.regex class. For example:

• The following regular expression matches any incident with a problem key that starts with MDS-5000.

  MDS-5000.*
  

• The following regular expression matches any problem with the text OutOfMemory. Because the regular expression is case-sensitive, it will not match problems with the text outofmemory.

  .*OutOfMemory.*
  

You can add and remove filters and get a list of filters or the detail of one filter using the DiagnosticConfig MBean.

Table 13-5 shows the operations and attribute for the configuring problem suppression filters and a description of each.

addProblemKeyFilter(".*OutOfMemory.*)

getProblemKeyFilters()

getProblemKeyFilter(id)

removeProblemKeyFilter(id)

To configure a problem suppression filter:

1. From the target navigation pane in Fusion Middleware Control, expand the farm, then WebLogic Domain.

2. Select the domain.

3. From the WebLogic Domain menu, choose System MBean Browser.

   The System MBean Browser page is displayed.

4. Expand Application Defined Beans, then oracle.dfw, then domain.domain_name, then dfw.jmx.DiagnosticsConfigMBean.

5. Select one of the DiagnosticConfig entries. There is one DiagnosticConfig entry for each server.

6. In the Application Defined MBeans pane, select the Operations tab.

7. Click addProblemKeyFilter. The Operation: addProblemKeyFilter page is displayed, as shown in the following figure:

   
   Description of the illustration problemkeyfilter.gif
   
   

8. For Value, enter a regular expression that represents a pattern that you want to match pattern. For example, in a development environment, you might want to add a filter so that incidents are not created when java.lang.IllegalStateException Java Exceptions are reported. In that case, enter the following:

   ".*[java.lang.IllegalStateException].*"
   

9. Click Invoke.

10. Click Return to return to the Application Defined MBeans page.

You can delete the filters using the removeProblemKeyFilter operation.

You can retrieve a specific filter, passing the ID of the filter to the getProblemKeyFilter operation.

Alternatively, you can retrieve a list of the filters using the getProblemKeyFilters attribute:

1. From the target navigation pane, expand the farm, then WebLogic Domain.

2. Select the domain.

3. From the WebLogic Domain menu, choose System MBean Browser.

   The System MBean Browser page is displayed.

4. Expand Application Defined Beans, then oracle.dfw, then domain.domain_name, then dfw.jmx.DiagnosticsConfigMBean.

5. Select one of the DiagnosticConfig entries. There is one DiagnosticConfig entry for each server.

6. In the Application Defined MBeans pane, select the Attributes tab.

7. Click ProblemKeyFilters.

   The list of problem suppression filters is displayed.

13.3.4 Configuring WLDF Watch and Notification for the Diagnostic Framework

Fusion Middleware configures a WLDF Diagnostics Module that contains a set of Watch and Notification rules for detecting a specific set of critical errors and creating an incident for each occurrence of those errors. The module is called Module-FMWDFW and contains the following set of Watch conditions:

The Diagnostic Module also includes a configured WLDF JMX Notification FMWDFW-notification of type oracle.dfw.wldfnotification. You can reuse this WLDF JMX Notification for your own WLDF Watch conditions to create an incident:

1. Display the Administration Console, as described in Section 3.4.1.

2. In the left pane, expand Diagnostics and select Diagnostic Modules.

   The Summary of Diagnostic Modules page is displayed.

3. Click Module-FMWDFW.

   The Settings for Module-FMWDFW page is displayed.

4. Select the Watches and Notifications tab, which is shown in the following figure:

   
   Description of the illustration dfw_notif.gif
   
   

5. Select the Watches tab and click New.

   The Create Watch page is displayed.

6. For Name, enter a name for the watch.

   You can enter any name. Alternatively, you can use the following format to force the Diagnostic Framework to use a custom message ID:

   message-id#[application_name]#any_text
   

   The message ID consists of a prefix that can be 1 to 6 characters, and a number, that can be 1 to 6 digits. The application name is optional. For example:

   SOA-40500#My_Watch_Name
   

   The following example uses the application name soa_infra:

   SOA-40501#soa-infra#My_Watch_Name
   

   The Diagnostic Framework uses the message ID as the incident message ID in constructing the incident problem key.

7. For Watch Type, select a type, for example, Server log.

8. Click Next.

9. For Current Watch Rule, construct an expression. For example, to construct the expression (SEVERITY = 'Error') AND (MSGID = 'BEA-000337'):

   1. Click Add Expressions.

   2. For Message Attribute, select Severity.

   3. For Operator, select =.

   4. For Value, enter ERROR.

   5. Click OK.

   6. Click Add Expressions.

   7. For Message Attribute, select MSGID.

   8. For Operator, select =.

   9. For Value, enter BEA-000337.

   10. Click OK.

   11. In the Create Watch page, ensure that the operator selected is AND.

   12. Click OK.

10. Click Next.

11. Select an alarm type and click NEXT.

12. For Notifications, select FMWDFW-notification and move it to the Chosen box.

13. Click Finish.

For more information on creating watches, see "Construct watch rule expressions" in the Administration Console Online Help.

13.4.1 Roadmap—Investigating, Reporting, and Resolving a Problem

Typically, investigating, reporting, and resolving a problem begins with a critical error. This section provides an overview of that workflow.

Figure 13-4 illustrates the tasks that you complete to investigate, report, and resolve a problem.

Figure 13-4 Flow for Investigating a Problem

Description of "Figure 13-4 Flow for Investigating a Problem"

The following describes the workflow illustrated in Figure 13-4:

1. You notice that the system, component, or application is not functioning as expected. For example, you notice that there is a performance problem or users have reported that the application that they are trying to access is reporting errors.

2. Check to see if a problem and an incident have been created that may be related to the symptoms you are observing:

   1. View the set of problems by using the WLST listProblems command, as described in Section 13.4.2.1.

   2. If a problem has been created, list the incidents related to the specific problem using the listIncidents command, as described in Section 13.4.2.2.

3. If an incident has not been created, go to Step 4. If an incident has been created, go to Step 5.

4. If you do not see any incidents listed that are related to your problem, you can create an incident manually using the createIncident command to capture diagnostics for the problem.

   Consider creating an incident when you encounter an issue, such as software failure or performance problem, and you want to gather more diagnostic data. You can view the log files and the messages in the files. If there is a specific message that you believe is related to the issue you are seeing, you can use the message ID in the createIncident command.

   See Section 13.4.6.1 for more information about creating an incident.

5. View the details of the specific incident using the showIncident command, as described in Section 13.4.2.2. This command lists information about the incident, including the related message ID, the time of the incident, the ECID, and the files generated by the incident.

6. Use the getIncidentFile command to view the contents of files for the incident, as described in Section 13.4.2.2. The contents may provide information to guide you to the source of the problem and help in resolving it.

7. If the contents of the files for the incident do not help you to resolve the problem, you can execute additional dumps to view detailed diagnostics. For example, if you are experiencing performance problems, execute the dms.metrics dump. See Section 13.4.4 for information about the dumps available and how to execute them.

8. If you still cannot resolve the problem, package the incident, along with the RDA report, and send them to Oracle Support. See Section 13.4.6.2 and Section 13.4.7 for information about packaging incidents and generating RDA reports.

13.4.2 Viewing Problems and Incidents

You can view the set of problems, the list of incidents, and the details of a particular incident using the WLST command-line utility, as described in the following topics:

• Viewing Problems

• Viewing Incidents

listProblems([adrHome] [,server])

listProblems()
Problem Id      Problem Key
        1       BEA-101020 [HTTP]

listIncidents([id], [ADRHome])

listIncidents()
Incident Id     Problem Key              Incident Time
        2       BEA-101020 [HTTP]        Fri Feb 26 13:42:01 PDT 2010
        1       BEA-101020 [HTTP]        Tue Feb 23 06:17:39 PDT 2010

listIncidents(id='1')
Incident Id     Problem Key              Incident Time
        2       BEA-101020 [HTTP]        Fri Feb 26 13:42:01 PDT 2010
        1       BEA-101020 [HTTP]        Tue Feb 23 06:17:39 PDT 2010

showIncident(id, [adrHome] [,server])

showIncident(id='1')
Incident Id: 1
Problem Id: 1
Problem Key: BEA-101020 [HTTP]
Incident Time: Tue  Feb 23 06:17:39 PDT 2010
Error Message Id:  BEA-101020
Execution Context: 0000IExqUvyAhKB5JZ4Eyf1Afdj600009i
Flood Controlled: false
Dump Files :
    dms_ecidctx1_i1.dmp
    jvm_threads2_i1.dmp
    dms_metrics3_i1.dmp
    odl_logs4_i1.dmp
    odl_logs5_i1.dmp
    diagnostic_image_AdminServer_2010_02_23_06_17_42.zip
    readme.txt

getIncidentFile(id, name [,outputFile] [,adrHome] [,server])

getIncidentFile(id='1', name='odl_logs4_i1.dmp', outputFile='/tmp/odl_logs4_i1_dmp.output')

13.4.3 Analyzing Specific Problem Keys

The Diagnostic Framework provides a set of well-defined problem keys for unhandled exceptions. These exceptions are either detected through the existing WLDF Watch "UncheckedException" or through the Diagnostic Framework java.lang.Thread.UncaughtExceptionHandler uncaught exception handler. Previously, the Diagnostic Framework generated problem keys with different formats for the same type of issues. Table 13-6 describes these problem keys and how to use them to investigate a problem.

13.4.4 Working with Diagnostic Dumps

If you suspect a problem, you can make use of the built-in diagnostic dumps to report detailed diagnostics that can help diagnose the problem. Diagnostic dumps provide a means to output and record diagnostics data which serve as valuable information when diagnosing issues with Oracle Fusion Middleware components, applications, and infrastructure. The output from these dumps is intended to be used by customers and Oracle Support to diagnose issues with Oracle Fusion Middleware.

Diagnostic dumps are executed in the following ways:

• Manually, using WLST commands, as described in the following sections

  For example, if your Java EE application is hanging and you suspect a deadlock, you could use the jvm.threads dump to obtain the set of threads.

• Automatically, when the Diagnostic Framework detects a critical error and creates an incident or when the administrator creates an incident

listDumps([appName] [,server])

listDumps(server='soa_server1')
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root. 
For more help, use help(domainRuntime)

odl.activeLogConfig
jvm.classhistogram
dms.ecidctx
wls.image
odl.logs
dms.metrics
odl.quicktrace
http.requests
jvm.threads

Use the command describeDump(name=<dumpName>) for help on a specific dump.

describeDump(name='dms.metrics')
Name: dms.metrics
Description: Dumps DMS (Dynamic Monitoring Service) metrics.
Mandatory Arguments: 
Optional Arguments:
    Name        Type     Description
    format      STRING   Format of the dump output; raw or xml

executeDump(name='dms.metrics', outputFile='/tmp/dumpout.txt', id='1')
Dump file dms_metrics1_i1.dmp added to incident 1

13.4.5 Configuring and Using Diagnostic Dump Sampling

Diagnostic dump sampling captures the output of diagnostic dumps at specified intervals. By sampling at regular intervals, diagnostic dump sampling can help to reveal issues such as slow running web requests, and where work is being performed in those requests.

This section contains the following topics:

• Understanding Diagnostic Dump Sampling

• Configuring Dump Sampling

• Listing Dump Samplings

• Retrieving the Dump Sampling Output

dfw_dumpArchivennn.Sampling_Name.{txt]|zip}

dfw_dumpArchive17394218037.JVMThreadDump.txt

dfw_dumpArchivennn.readme.txt

addDumpSample(sampleName="sample_name", diagnosticDumpName="dump_name",
 [appName="application_name",] samplingInterval=num_seconds,
 rotationCount=num_samples, [dumpedImplicitly={true|false},]
 [toAppend={true|false},] [args={"arg_name" : "value"},] 
 [server="server_name"])
 

addDumpSample(sampleName="HTTPSampling", diagnosticDumpName="http.requests", samplingInterval=300, rotationCount=10, server="soa_server1")

HTTPSampling is added

updateDumpSample(sampleName="sample_name",  
[        appName="application_name",] samplingInterval=num_seconds, 
         rotationCount=num_samplings, [dumpedImplicitly={true|false},] 
[        toAppend={true|false},] [args={"arg_name" : "value"},] 
[        server="server_name"])
 

updateDumpSample(sampleName="HTTPSampling", samplingInterval=200,
                   rotationCount=5, server="soa_server1")

HTTPSampling is updated

removeDumpSample(sampleName="sample_name", [server="server_name"])
 

removeDumpSample(sampleName="HTTPSampling", server="soa_server1")

Removed HTTPSampling

enableDumpSampling(enable={true|false}, [server="server_name"])

enableDumpSampling(enable=false)

isDumpSamplingEnabled([server="server_name"])

listDumpSample([sampleName="sample_name",] [server="server_name"])
 

listDumpSamples(server="soa_server1")
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root. 
For more help, use help(domainRuntime)
 
Name              : JavaClassHistogram
Dump Name         : jvm.classhistogram
Application Name  : 
Sampling Interval : 1800
Rotation Count    : 5
Dump Implicitly   : false
Append Samples    : true
Dump Arguments    : 
 
Name              : JVMThreadDump
Dump Name         : jvm.threads
Application Name  : 
Sampling Interval : 60
Rotation Count    : 10
Dump Implicitly   : true
Append Samples    : true
Dump Arguments    : context=true, timing=true

executeDump(name="dfw.samplingArchive",outputFile="filename" 

executeDump(name='dfw.samplingArchive', args={'sampleName' : 'JavaClassHistogram'})

getSamplingArchives([sampleName="sample_name"] [,outputFile="filename" [,server="server_name"])

getSamplingArchives(sampleName="JavaClassHistogram", outputFile="/tmp/sampling.zip")
 

unzip -l /tmp/sampling.zip
Archive:  /tmp/sampling.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
  6241768  07-27-12 11:19   dfw_samplingArchive8680976839106379444.JavaClassHistogram.dmp
      552  07-27-12 11:19   dfw_samplingArchive7861027727509995202.readme.txt
 --------                   -------
  6242320                   2 files

13.4.6 Managing Incidents

The Diagnostic Framework stores incidents, whether they are created automatically or manually, and Oracle Fusion Middleware provides tools to help you process incident reports and to package those incidents to send to Oracle Support. The following sections describe:

• Creating an Incident Manually

• Packaging an Incident

• Generating an RDA Report

• Purging Incidents

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

purge -age 60 

13.4.7 Generating an RDA Report

You can use the Remote Diagnostic Agent (RDA), a command-line diagnostic tool, to provide a comprehensive picture of your environment. Additionally, RDA can provide recommendations on various topics, for example configuration and security. This aids you and Oracle Support in resolving issues.

RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. A security filter is provided if required.

For more information about RDA, see the readme file, which is located at:

(UNIX) ORACLE_HOME/rda/README_Unix.txt
(Windows) ORACLE_HOME\rda\README_Windows.txt

13.5.1 Understanding the Health Test Framework

The Health Test Framework lets you execute diagnostic tests and collects the results into detailed diagnostic reports. The diagnostics tests are installed when you install or patch Oracle Fusion Middleware.

You can use the Health Test Framework to check normal system health and to troubleshoot system problems. You can configure your Oracle Fusion Middleware environment to run the diagnostic tests using the Health Test Framework command line interface.

A diagnostic test may or may not be associated with a particular error message. If an Oracle Fusion Middleware application handles a particular error in a way that triggers the creation of an incident, then any diagnostic tests that are associated with the error message for the incident run automatically. The test results are associated with the incident and the identity of the user who received the error message is recorded. The diagnostic tests and results are stored in file-based repository, as described in Section 13.5.2.

13.5.2 Understanding the Health Test Framework File Repository

The Health Test Framework stores tests and test results in a file-based repository. The repository has two primary stores:

• TestDef, which contains test definitions, test metadata, data related to test parameters, language data files, and index files.

• Test Run, which contains data related to the test run and execution, report files, and index files.

Index files are used for improving the speed of searching and querying.

The repositories are created when you register tests or run tests if the repository does not exist at the specified location.

You can specify the location of the repository in the configuration properties file, diagfsconfig.properties. This file is located in:

MW_HOME/diagbase/dtf_filestore

By default, the file stores are located in:

MW_HOME/diagbase/dtf_filestore/testdef
MW_HOME/diagbase/dtf_filestore/testrun

13.5.3 Using the Health Test Framework Command Line

The Health Test Framework provides two command line interfaces, dfwhealthtestadminctl.sh for administration commands and dfwhealthtestctl.sh for execution commands, as described in the following sections:

• dfwhealthtestadminctl.sh Command Line

• dfwhealthtestctl.sh Command Line

ORACLE_COMMON_HOME/common/bin

/dfwhealthtestadminctl.sh command [options]

help command

register testfile=test_xml_files | dir=test_dirs
          [validateonly={Y|N]]
          testjar=jar_file_location

index [refresh={testdef|testrun}

13.5.3.2 dfwhealthtestctl.sh Command Line

The Health Test Framework provides the dfwhealthtestctl.sh command line for running the health tests. With it, you can run tests, list runs and tests, get a description of a test, get the status of tests, and retrieve a report of a test or test run.

The dfwhealthtestctl.sh command-line interface is located in:

ORACLE_COMMON_HOME/common/bin

The syntax of the command-line interface is:

/dfwhealthtestctl.sh command [options]

Table 13-10 lists the Health Test Framework dfwhealthtestctl.sh commands:

Table 13-10 Health Test Framework dfwhealthtestctl.sh Commands

Command	Description
desctest	Provides a detailed description of a specified test.
listrun	List the test runs.
listtest	Lists the description of one or more tests.
report	Extract a report of a test run.
run	Run one or more tests.
status	Get the status of a test run or execution.

13.5.3.2.1 desctest

Provides a detailed description of a specified test.

The syntax is:

desctest {testid=testid | testname=testname}]
          [showparam={Y|N}]

The following table describes the parameters for the desctest command:

Parameter	Description
testid	The ID of the test. To obtain the ID, use the listtest command. You must specify either this parameter or the testname parameter.
testname	The name of the test. You must specify either this parameter or the testid parameter.
showparam	Optional. A Boolean value to specify whether to retrieve the parameters for the test. Value values are Y or N.

13.5.3.2.2 help

Provides help for the commands.

The syntax is:

help command

The following table describes the parameters for the help command:

Parameter	Description
command	The name of the command.

13.5.3.2.3 listrun

Retrieves summary information for the test runs. You can retrieve the information by run name, run status, start time, and test name.

The syntax is:

listrun { runname=runname | testname=testname | status=status | lasthours=hours }
         [showexec={Y|N]]

The following table describes the parameters for the listrun command:

Parameter	Description
runname	The name of the run for which you want to return results. To get the runname, use the listrun command without any parameters.
testname	The name of the test for which you want to return results.
status	The status of the run for which you want to return results. Valid values are running, warning and success.
lasthours	The number of hours for which you want to return results.
showexec	Optional. A Boolean value to specify whether to retrieve the execution of the tests. Value values are Y or N.

13.5.3.2.4 listtest

Lists one or more tests. You can use wildcards to specify the names of the tests

The syntax is:

listtest [testname=testname]
          [productcode=productcode

The following table describes the parameters for the help command:

Parameter	Description
testname	The names of one or more tests, in a comma-separated list. You can use wildcards to specify a pattern to match the names of tests.
productcode	The name of the product. You can use wildcards to specify a pattern to match the name of the product.

13.5.3.2.5 report

Extract a test report for a particular test run or execution.

The syntax is:

report { runname=runname | runid=run_id | execid=execution_id }
          [destdir=destination_directory}
          [format=HTML | XML]
          [translate={Y | N}

The following table describes the parameters for the report command:

Parameter	Description
runname	The name of the run. To get the name, use the listrun command. You must specify this parameter, runid, or execid.
runid	The ID of the run. To get the ID, use the listrun command. You must specify this parameter. runname, or execid.
execid	The execution ID of the run. To get the ID, use the listrun command You must specify this parameter, runname, or runid.
destdir	Optional. The destination directory to which the report files are written. If not specified, the report files are extracted under the java.io.tmpdir/user.name/diagfwk directory, where java.io.tmpdir and user.name are Java system properties.
format	The format of the generated report. Valid values are HTML and XML.
translate	A Boolean value specifying whether to translate the report. Value values are Y or N.

13.5.3.2.6 run

Run the specified tests.

run {test=testnames | productcode=codes }
     [runname=name
     [runoption=asynch]
     [input:param1=value1 param2=value2 ...
     [inputfile=filename] 
     [contextfile=context_file] 
     [moninterval=monitoring_interval] 
     [nthreads=number_of_threads] 
     [reportshowparam=show_param] 

The following table describes the parameters for the run command:

Parameter	Description
test	The names of one or more tests, in a comma-separated list. If you specify a single test name, you can also specify one or more input parameter names and values using the input parameter. You must specify this parameter or the productcode parameter.
productcode	The code for one or more products that include the tests you want to run. Specify more than one product code in a comma-separated list.
runname	Optional. A name for the current run. If not specified, the command generates a default run name.
moninterval	The interval, in seconds, in which the status of the run is uploaded to the test repository. The default value is 30 seconds.
nthreads	The number of parallel threads that should be spawned to execute tests in this run. The default value is 5. Specifying a value of 1 for this parameter will execute the tests serially.
runoption	Optional. Options for running the tests. The valid value is asynch, which runs the test asynchronously in the background.
input	Optional. If you use this parameter, use one or more parameter name/ value pairs. For example: domainhome=/scratch/oracle/domains/basedomain You can only use this parameter when you run only one test.
inputfile	The path to a file containing input parameters. The file should have each input name/value pair on a separate line. The format of each line is: input_param_name:input_param_value
contextfile	The path to a file that contains user system properties to be used at run time.
reportshowparam	Sets the input and output parameter display. By default all parameters are hidden during execution. To specify that the parameters are shown, use input or output, or a comma-separated string with both. For example: reportshowparam=input,output

13.5.3.2.7 status

Get the status of a test run by specifying the run name, run ID, or execution ID.

The syntax is:

status { runname=runname | runid=run_id | execid=execution_id }
      [printtree={Y|N}

The following table describes the parameters for the status command:

Parameter	Description
runname	The name of the run. To get the run name, use the listrun command.
runid	The ID of the run. To get the ID, use the listrun command.
execid	The execution ID of the run. To get the ID, use the listrun command.
printtree	Specifies whether to print the status of all nested conditions. Valid values are Y and N.

13.5.4 Managing the Health Test Framework

You can register tests and rebuild the index, as described in the following topics:

• Creating a Repository and Registering Health Test Framework Tests

• Rebuilding the Health Test Framework Indexes

dfwhealthtestadminctl.sh index refresh=testdef 

13.5.5 Running Health Test Framework Diagnostic Tests

You run the Health Test Framework diagnostic tests using the dfwhealthtestctl.sh command line interface, which is located in:

ORACLE_COMMON_HOME/common/bin

To run a test, use the run command, using the format described in Section 13.5.3.2.6. For example, to run the test SampleTest, use the following command:

./dfwhealthtestctl.sh command
               [testfile=SampleTest.xml] 

You can run one or more tests at the same time:

• To run one or more tests by specifying the test names in a comma-separated list:

  ./dfwhealthtestctl.sh run test=oracle.dfw.healthtest.test.sample.SampleTest2
  Processing "run" command ...
   
  Executing run ID "1340125981427" with name "TestRun_1340125981427" ...
  sampleTest2 sleep for a while....
  sampleTest2 done
   
  Run ID "1340125981427" with name "TestRun_1340125981427" started at 06/21/2012
   10:13:01 AM and completed at 06/21/2012 10:13:04 AM. No diagnostic issues were detected.
  

• To run one or more tests by using wildcards for the test name:

  ./dfwhealthtestctl.sh run test=oracle.dfw.healthtest.test.sample.SampleTest*
  

• To run all tests associated with a particular product:

  ./dfwhealthtestctl.sh run productcode=idm
  

If you are running only one test, you can specify input options either on the command line or in an input file:

• To specify the input options on the command line, use the input: parameter:

  ./dfwhealthtestctl.sh run test=oracle.dfw.healthtest.test.sample.SampleTest,oracle.dfw.healthtest.test.sample.SampleTest2
                             input:userid=11 input:roleid=22
  

• To specify the input options in an input file, create the file and specify it on the command line using the inputfile parameter. The following shows the format of the input file:

  input_parameter1:parameter_value1
  input_parameter2:parameter_value2
  

  For example:

  userid:11
  roleid:22
  

  Then, specify the file on the command line:

  ./dfwhealthtestctl.sh run test=oracle.dfw.healthtest.test.sample.SampleTest
                             inputfile=/tmp/inputfile
  

13.5.6 Searching for Health Test Framework Diagnostic Tests

You can search for tests or runs using the dfwhealthtestctl.sh listtest command. Optionally, you can specify the name of the test or product code:

• To search for all tests:

  ./dfwhealthtestctl.sh listtest
  

• To search for one test named SampleTest:

  ./dfwhealthtestctl.sh listtest testname=SampleTest
  

• To search for more than one test using wildcards:

  ./dfwhealthtestctl.sh listtest testname="S*"
  

• To search for all tests related to a particular product:

  dfwhealthtestctl.sh listtest productcode=idm
  

13.5.7 Retrieving a Description of a Health Test Framework Test

You can retrieve the description of a specified test using the dfwhealthtestctl.sh listtest command. For example, to get the description of the test SampleTest, along with its parameters, use the following command:

./dfwhealthtestctl.sh desctest testname="SampleTest" showparam=Y

13.5.8 Listing Health Test Framework Test Runs

You can query the results of test runs, by test name, run name, status, or time, using the dfwhealthtestctl.sh listrun command:

• To query the results for a test run by specifying the test name:

  ./dfwhealthtestctl.sh listrun testname="SampleTest"
  

• To query the results for a test run by specifying the run name:

  ./dfwhealthtestctl.sh listrun runname="run_1" 
  

• To query the results for a test run by specifying the status. For example, to get the results of all tests with a status of running:

  ./dfwhealthtestctl.sh listrun status=r 
  

• To query the results for test runs that were started in the last 2 hours:

  ./dfwhealthtestctl.sh listrun lasthours=2 
  

13.5.9 Generating Health Test Framework Reports

You can generate reports of the test runs for the Health Test Framework using the dfwhealthtestctl.sh report command. You can generate HTML or XML reports.

You can specify the run name, run ID or execution ID>

For example, to generate a report in HTML format for the run with the run ID of 1330128064268:

dfwhealthtestctl.sh report runid=1330128064268 format=HTML