1. From the Targets menu, select Middleware.

   Oracle Enterprise Manager Cloud Control displays the Middleware page that lists all the middleware targets being monitored.

2. On the Middleware page, from the Add list, select Oracle Fusion Middleware / WebLogic Domain and click Go.
3. On the Find Targets page, specify the Administration Server Host, Port, Username, Password, and Agent (local or remote) details.

   Figure 7-1 New Domain Discovery

   
   

   

   
   

   In the Advanced section, select the JMX Protocol from the list. By default, the Discover Application Versions appears checked which enables administrators to discover all versions of deployed SOA Composites. However, if you uncheck this option, then you can discover only the latest default version of SOA composites.

   Note:

   When the SOA Infrastructure application is down, if you uncheck the Discover Application Versions check box, then, only composites with single version is discovered. If there are composites with multiple versions, they are ignored.

   Figure 7-2 Upgrade Domain Discovery

   
   

   

   
   

   Note:

   • If you have targets which were discovered with the Discover Application Versions box checked (which is the default, see Figure 7-1), but now want to disable this option, perform the following steps:

     • Go to the WebLogic Domain target page.

     • On the Monitoring Configuration page, update the value of Discover application versions to false. (See Figure 7-2.)

     • Perform a domain refresh.

     Doing this will discover new composite targets (without any version numbers in their names) that will not contain the metric history from the previous targets.

   • Once you are in a state where you have composite targets without version numbers in their names, if you add more SOA composite versions, the version specified as the default version in the SOA Suite will be monitored. Historical metrics will be retained on the same target whenever the default version changes.

   Click Continue.

4. You will return to the Middleware page. You will see the SOA instances under the WebLogic Domain.

   Note:

   SOA Composites that are created after the discovery of SOA Suite Domain are not displayed automatically. To view all the SOA Composites, navigate to the Home page of the WebLogic Server target and refresh the WebLogic Domain.

   To refresh the domain manually, see My Oracle Support note 1586853.1.

   To enable Automatic Refresh of the domain, see My Oracle Support note 1531733.1.

   Note:

   For a successful monitoring of SOA 12.2.1.1 and above targets, you must have Enterprise Manager Cloud Control 13.2 PG Release and Agent with 13.2 PG Plug-in. Monitoring of SOA 12.2.1.1 and above versions is not compatible using older Enterprise Manager or Agent versions. Metrics related to SOA Composite entities will not be collected due to agent side dependency and hence, SOA Composite entities will not be discovered. Features that require SOA Composite as input parameter will not work.

1. From the Targets menu, select Middleware.
2. On the Middleware page, click the target you are interested in. For example, SOA Infrastructure.
3. On the Home page of the target, from the target-specific menu, select Target Setup, and click Verification.
4. On the Target Setup Verification page, in the Functionality Check section, click Scan.
5. If setup problems are detected, repair them. See Repairing Target Monitoring Setup Issues.

1. From the Targets menu, select Middleware.
2. On the Middleware page, click the target you are interested in. For example, SOA Infrastructure.
3. On the Home page of the target, from the target-specific menu, select Target Setup, and click Verification.
4. On the Target Setup Verification page, in the System Check section, click Scan.

   Note:

   • The Availability(%) column shows the availability rate of the target types. Click the rate to drill down and view more details.

   • The Time since last collection (min) column shows the total time elapsed since the last metric collection for the target.

1. Run functionality-level diagnostic scan to identify setup-related issues. See Running Functionality-Level Diagnostic Checks.
2. If setup issues are found in the Functionality Check section, click the repair icon against the functionality that requires to be fixed.

   In the dialog that appears, enter the required details, and click Re-Scan and Save to validate the credentials, run the functionality check again, and save the details in Enterprise Manager. If you are sure the credentials are correct, then click Save to save the details without running the check again.

   Note:

   • The host credentials you are expected to provide are credentials of the host where the Management Agent, which monitors the SOA Infrastructure, is running.

   • While repairing dehydration store issues, you are required to provide SOA dehydration store configuration details such as the database system and the SOA repository credentials.

     If the configuration information has been collected, and if the association between the database system and the SOA infrastructure already exist, then the database system is pre-populated by default, and you need to enter only the credentials of the SOA repository. Otherwise, click the torch icon and manually select the database system with which the SOA infrastructure communicates, and enter the credentials of the SOA repository.

     The connection descriptor is pre-populated by default is an editable field, and appears in multiple rows if it is an Oracle RAC database. Do not modify the descriptor unless you want to correct it.

     The data source type is displayed only if the database system is an Oracle RAC database. The data source type can be either Multi Data Source or GridLink Data Source. Also note, that the data source type appears as 'NA' if details of the database system do not match with the connection descriptor.

An Integration Workload Statistics (IWS) report contains various statistics, depending on the data collection level that you have set. In addition to system-wide resource usage data, the report can include service and reference endpoint statistics, BPEL and EDN backup queue statistics, and BPEL instance statistics. Statistics on BPEL activities may also be included.

The IWS report contains the following broad sections when the data collection level is set to finest:

1. From the SOA Infrastructure menu, select Diagnostics and then click Generate IWS Report.
2. Click IWS Settings.
3. Set the IWS Collection to ON.
   OFF disables IWS data collection until it is manually set to ON again.
4. Select a Snapshot Interval in minutes.
   The snapshot interval is the periodic interval at which data snapshots are collected.
5. Select a Data Collection Level. The level selected determines the metrics that are collected.
   Use the Minimum level to collect only system-wide resource usage data. The Basic level additionally includes service and reference endpoint statistics, BPEL and EDN backup queue statistics, and BPEL instance statistics. If you choose Normal, it includes additional statistics on BPEL activities like Receive, Invoke, Pick, and onMessage. The Finest level additionally includes data on all BPEL activities.
6. Click Save Changes to save your configuration changes.

1. From the SOA Infrastructure menu, select Diagnostics, and then click Generate IWS Report.
2. Select the period for which you wish to generate a report. Select timestamps for Start Date and End Date.
   Ensure that the time period does not span server restarts, or periods where you have disabled IWS by setting Data Collection Level to OFF.
3. Click the appropriate Report Format to generate and download the report.
   You can choose between HTML, XML formats, and CSV (comma-separated values).
4. Optionally, choose a partition name if you are using composite partitions and wish to limit your report to a particular partition.
   The Select Composites field is displayed. This option enables you to select from all composites in the selected partition.
5. Under Select Composites, optionally choose one or more composite names to restrict your report to the specified composite applications.
6. Click Generate Report.

1. Set the host and the WebLogic Domain preferred credentials using Target Setup Verification. For details, see Configuring the SOA Suite with Target Verification.
2. To view the state of the listed SOA instances, enable the Capture Composite State flag on the instance tracing page as follows:
   1. On the SOA Infrastructure home page, from SOA Infrastructure menu, select Fusion Middleware Control.
   2. Navigate to the home page of the SOA Infrastructure target.
   3. From SOA Infrastructure menu, select SOA Administration, and then click Common Properties.
   4. On the SOA Infrastructure Common Properties page, select the Capture Composite Instance State check box.

Select the appropriate search link based on the version of your SOA target:

• Instance Tracing for SOA 11g Targets

• Instance Tracing for SOA 12c Targets

1. In Cloud Control, from the Targets menu, select Middleware.
2. On the Middleware page, click the target you are interested in. For example, SOA Infrastructure.
3. From the SOA Infrastructure menu, select Trace Instance.
4. On the Instance Tracing page, perform instance search. To do so, see Table 7-5.
5. To trace an instance across composites, do the following:

   • For a SOA 12c target, click Flow Instance ID.

   • For a SOA 11g target, click Composite Instance ID.

   You can further drill down to the component audit trail by clicking the component instance available in the trace table.

6. Click OK.

1. In Cloud Control, from the Targets menu, select Middleware.
2. On the Middleware page, click the target you are interested in. For example, SOA Infrastructure.
3. From the SOA Infrastructure menu, select Trace Instance.
4. On the Instance Tracing page, perform instance search. To do so, see Table 7-5.
5. Select an instance, and Click Trace.
6. In the Trace Instance dialog box, click Add to add the other SOA Infrastructure targets where this SOA instance has been executed.
7. In the Search and Add targets dialog box, select the other SOA Infrastructure targets, and click Select.
8. Click Set to set the WebLogic Domain Credentials, and Host Credentials if you haven't already set them.
9. Click OK. A flow trace job is scheduled to run immediately to collect the instance trace data across domains. On completion, a status is displayed in Trace Job Status column. Click the status link to drill down to the Flow Trace page.
10. Click OK.

1. From the Targets menu, select Databases to check if the database target representing the SOA Dehydration Store has been discovered in Enterprise Manager.
2. Check if at least one configuration for the SOA Infrastructure and WebLogic Server targets is available.
3. Navigate to the Target Verification page to run the functionality check for dehydration store. For more details, see Configuring the SOA Suite with Target Verification.

1. From the Targets menu, select Middleware, then click on a SOA Infrastructure target.
2. In the SOA Infrastructure Home page, click the Dehydration Store Performance tab.
3. The following details area displayed:

   • Throughput indicators that provide details of the general health of the database instance.

   • Wait bottleneck issues related to the CPU, I/O, and Wait events.

   • Database diagnostics by clicking the SQL ID in the Top SOA SQL table.

   • JVM diagnostics by clicking the JVM Diagnostics link for the respective SQL ID.

   • Tablespace utilization for the SOA schema.

   • Performance data recorded by the ASH.

   • Key SOA tables and tablespace details related to the SOA schema.

1. Ensure that you set the SOA Database Host Credentials and SOA Database user Credentials before scheduling a SOA diagnostics job.
2. From the Targets menu, select Middleware.
3. On the Middleware page, select a SOA Infrastructure target. The SOA Infrastructure home page is displayed.
4. From the SOA Infrastructure target menu, select Diagnostics, then click Schedule SOA Diagnostics Job.
5. In the General section, enter a name and description for the job.
6. In the Target section, select a database instance from the table. To add an instance, click Add. From the target selector dialog box, select a database instance, and click Select.
7. In the Parameters section, enter the following details:

   • Report Time Period is the period for which you want to collect the diagnostic data. This is a mandatory field. By default, data for last one week is collected.

   • Optionally, you can select a desired value for System Backlog Report.

   • To get details about open instances, completed instances, or rolled back instances for a product, you must choose the Instance Growth Report.

   • To get a report on invoke process delays, callback delays, callback processing delays, select BPEL Execution Report, and BPEL Performance Report

   • To understand invoke delays, and engine time better, select Mediator reports like Mediator Execution Report, and Mediator Performance Report.

   • To understand pending events in an event queue, select EDN Report.

   • To get a summary of all the faults, select Fault Summary Report and Detailed Fault Report.

   • To view the human workflow tasks, select Human Workflow Report.

   • To receive a SOA Diagnostic report through an email, select Email Notification.

   • Subject, enter a subject for your email.

   • E-mail To, add contacts to whom this report must be sent.

   • E-mail Cc, add contacts who must be copied on the diagnostics report email.

8. In the Credentials section, provide the SOA Infra Dehydration Store user Credentials, and host credentials for the SOA Dehydration Store.
9. In the Schedule section, you can choose to either run job once or repeatedly. You can additionally schedule to run the job immediately or at a later point.
10. The Access table gives a summary of all the users and roles who have access to this job.
11. Click Submit.

1. Ensure that you set the SOA Database Host Credentials and SOA Database user Credentials before scheduling a SOA diagnostics job.
2. From the Targets menu, select Middleware.
3. On the Middleware page, select a SOA Infrastructure target. The SOA Infrastructure home page is displayed.
4. From the SOA Infrastructure target menu, select Diagnostics, then click All SOA Diagnostics Job.

   This page displays all the diagnostics jobs that have run already, and are scheduled to run.

The following are the types of SOA composite application faults displayed in Enterprise Manager Cloud Control:

• Business: Application-specific faults that are generated when there is a problem with the information being processed (for example, a social security number is not found in the database).

• System: Network errors or other types of errors such as a database server or a web service being unreachable.

• Oracle Web Service Manager (OWSM): Errors on policies attached to SOA composite applications, service components, or binding components. Policies apply security to the delivery of messages.

The following are the categories of SOA composite application faults in Enterprise Manager Cloud Control:

• Recoverable

  • Business faults and some specific system faults

  • Oracle Mediator input file path and output directory mismatch

  • An Oracle BPM Worklist user is not authorized to perform relevant (expected) actions

• Nonrecoverable

  • Rejected messages

  • Most system faults

  • Non-existent references

  • Service invocation failures

  • Policy faults

• Rejected Messages

  A fault is classified as a rejected message based on where it occurs. If a fault occurs before entering a SOA composite, without generating a composite instance, it is classified as a rejected message. A system or a policy fault can be identified as a rejected message.

Meet the following prerequisites before searching, viewing, and recovering SOA composite application faults:

Set the following as preferred credentials. These credentials can be set from the Target Setup Verification page. To do so, from the Targets menu, select Middleware. On the Middleware page, click the target you are interested in. For example, SOA Infrastructure. On the Home page of the target, from the target-specific menu, select Target Setup, and click Verification:

• Credentials of the host on which the SOA server is running.

• Administrator credentials of the Oracle WebLogic Domain.

1. Meet the prerequisites. See Prerequisites for Searching, Viewing, and Recovering Faults.
2. From the Targets menu, select Middleware.
3. On the Middleware page, click the SOA Infrastructure target.
4. On the SOA Infrastructure target page, click Faults and Rejected Messages.
5. In the Faults and Rejected Messages tab, set the search criteria. See Setting Search Criteria.
6. Click Search.
7. View the faults:

   • To know the total faults in the SOA infrastructure, see Total Faults in SOA Infrastructure, which is placed in the footer of the results table.

   • To know the number of faults displayed in the table (out of the total number of faults in the SOA infrastructure), see Displayed Faults, which is placed in the footer of the results table.

   • To view details of each fault, see the results table.

   • To hide or unhide columns in the table, from the View menu, select Columns, then select the column name you want to hide or unhide.

   • To filter or perform a fine search for a particular column, enter a search keyword in the textbox placed above the column header. See Filtering Displayed Search Results

     For example, to filter and list all faults related to the BPEL engine type, in the Engine Type column, type bpel.

   • To sort the fault details alphabetically, click the column header based on which you want to sort the details.

   • To find out the number of rows to which the search results have been restricted, see the note below the table.

     For example, the following note appears if the rows were restricted to 20.

     This table of search results is limited to 20 fault instances. Narrow the
     results by using the search parameters.

1. Meet the prerequisites. See Prerequisites for Searching, Viewing, and Recovering Faults.
2. From the Targets menu, select Middleware.
3. On the Middleware page, click the SOA Infrastructure target.
4. On the SOA Infrastructure target page, click Faults and Rejected Messages.
5. In the Faults and Rejected Messages tab, set the search criteria. See Setting Search Criteria.
6. Click Search.
7. In the table, select one or up to 5 faults at a time, and from the Recovery Options menu, select an appropriate recovery action that matches your requirement. For information on the recovery actions, see Overview of the Recovery Actions for Resolving Faults.
8. Enterprise Manager displays an informational message with one of the following mentioned to confirm whether or not it can submit the recovery job successfully. Click OK, and take the necessary action if required.

   • If you have selected more than 5 faults, then the recovery job is not submitted. Select 5 or fewer faults, and try again. Alternatively, select 5 or more, and try a bulk recovery. See Recovering Faults in Bulk.

   • If there are no recoverable faults, then the recovery job is not submitted.

   • If there are faults that are recoverable and not recoverable, then the recovery job is submitted only for recoverable jobs. You can track the recover job. See Tracking Bulk Recovery Jobs, and Viewing Their Results and Errors.

9. Perform Step (1) to Step (5) again to verify if the faults you selected for recovery still appear in the search results. If they do not appear, then the recovery operation for those faults has been successfully submitted.

1. Meet the prerequisites.
2. From the Targets menu, select Middleware.
3. On the Middleware page, select a SOA Infrastructure target.
4. On the SOA Infrastructure target page, from the SOA Infrastructure menu, select Fault Management, then select Bulk Recovery.
5. On the Bulk Recovery Jobs page, click Create Job.
6. On the Create Bulk Recovery Job, in the Composite section, enter the following details:

   • Select Initiating or Participating composite type from the menu.

   • Click Add to add additional composites for which faults must be searched. In the Search and Select dialog box, select all the targets that you want to add to the list, and click Select.

   • Click Remove to delete a composite.

   Note:

   You can add only up to 10 composites.

7. In the Time section, enter the suitable values in the following fields to filter out the faults that you want to recover: Instance Created From, Instance Created To, Instance Updated, Fault Time To, and Fault Time From.
8. In the Fault Details section, set the details of the faults you want to recover. To do so, see Setting Fault Details for Recovering Faults in Bulk.
9. In the Recovery Options section, set the recovery and batch parameters. To do so, see Setting Recovery and Batch Details for Recovering Faults in Bulk.
10. In the Job Parameters section, schedule the bulk recovery job. To do so, see Scheduling Bulk Recovery Jobs to Run Once or Repeatedly.
11. To verify the number of faults that will be recovered for the given criteria, click Estimate Faults.

    A pop-up appears informing you of the total number of faults in the SOA Infrastructure based on the criteria you have set. Based on the count, you can decide whether or not you want to proceed. If required, you can adjust the settings. For example, you can modify the fault time period.

12. Click Submit.
13. Track the status of the bulk recovery job. For more information, see Tracking Bulk Recovery Jobs, and Viewing Their Results and Errors.

1. From the Targets menu, select Middleware.
2. On the Middleware page, select a SOA Infrastructure target.
3. On the SOA Infrastructure target page, click Faults and Rejected Messages tab.
4. In the Faults and Rejected Messages tab, set the search criteria. To do so, see Table 7-7.
5. Click Search.
6. In the table, select one or more faults, and click Bulk Recover.
7. In the Navigate to Bulk Recovery wizard, select the details that you want to carry forward from the selected faults in the table to the Create Bulk Recovery page. Select one of more from the following list: Composite, Fault start time, Fault end time, and Error message for the fault, and click OK.
8. In the Composites section, the composite name and partition field is pre-populated with the values passed from the Faults and Rejected Messages tab. If you want to add additional composites that need recovery, then click Add. You can add only up to 10 composites.
9. In the Time section, if you have passed custom values for Faults Start Time and Fault End Time, then the Instance Created From and Instance Created To fields are also updated with the same values. You can change these values if required. However, if you select Last 1 Day, then all the faults that have occurred across instances in the last one day since the previous bulk recovery job was submitted are displayed.
10. In the Fault Details section, the Error Message field may appear pre-populated if you have passed error message attribute using the Navigate to Bulk Recovery dialog box. If not, you can update this section. For more information, see Setting Fault Details for Recovering Faults in Bulk.
11. In the Recovery Options section, set the recovery and batch parameters. To do so, see Setting Recovery and Batch Details for Recovering Faults in Bulk.
12. In the Job Parameters section, schedule the bulk recovery job. To do so, see Scheduling Bulk Recovery Jobs to Run Once or Repeatedly.
13. To verify the number of faults that will be recovered for the given criteria, click Estimate Faults.

    A pop-up appears informing you of the total number of faults in the SOA Infrastructure based on the criteria you have set. Based on the count, you can decide whether or not you want to proceed. If required, you can adjust the settings. For example, you can modify the fault time period.

14. Click Submit.
15. Track the status of the bulk recovery job. For more information, see Tracking Bulk Recovery Jobs, and Viewing Their Results and Errors.
16. Search for faults again (How?) to verify if the faults you selected for recovery still appear in the search results.

    If they do not appear, then the recovery operation for those faults has been successful.

1. Meet the prerequisites. Prerequisites for Searching, Viewing, and Recovering Faults.
2. From the Targets menu, select Middleware.
3. On the Middleware page, click the SOA Infrastructure target.
4. On the SOA Infrastructure target page, click Error Hospital.
5. In the Error Hospital tab, set the search criteria. To do so, see Table 7-9.
6. Click Search.
7. In the table, select one or more faults, and click Bulk Recover.
8. The composite section appears pre-populated with Composite, Composite type, and Fault Owner details. You cannot add more composites or edit this section.
9. In the Time section, details like Instance Created From and Instance Created to are picked up from the Error Hospital page. Additionally, if you had provided Fault Created From, Fault Created To, Instance Updated From and Instance Updated To values, then these values will also appear pre-populated on this page. If not, you can enter these values to refine your search.
10. In the Fault Details section, usually, one of the fault parameters appear pre-populated, by default, it is fault name. However, if you have grouped your Error Hospital Report by other categories, then those values are populated accordingly. To refine your search, you may update the other fields in this section. For more information, see Setting Fault Details for Recovering Faults in Bulk.
11. In the Recovery Options section, set the recovery and batch parameters. To do so, see Setting Recovery and Batch Details for Recovering Faults in Bulk.
12. In the Job Parameters section, schedule the bulk recovery job. To do so, see Scheduling Bulk Recovery Jobs to Run Once or Repeatedly.
13. To verify the number of faults that will be recovered for the given criteria, click Estimate Faults.

    A pop-up appears informing you of the total number of faults in the SOA Infrastructure based on the criteria you have set. Based on the count, you can decide whether or not you want to proceed. If required, you can adjust the settings. For example, you can modify the fault time period.

14. Click Submit.
15. Track the status of the bulk recovery job. For more information, see Tracking Bulk Recovery Jobs, and Viewing Their Results and Errors.
16. Search for errors again to verify if the errors you selected for recovery still appear in the search results.

    If they do not appear, then the recovery operation for those errors has been successful.

This section describes the following:

• Tracking Bulk Recovery Jobs, and Viewing Their Results and Errors

• Creating Bulk Recovery Jobs Using EMCLI and Web Services

This section covers the following examples:

• Running Bulk Recovery Job Every Night

• One Time Job with Specific Time Interval to Recover Faults

1. Meet the prerequisites. See Prerequisites for Searching, Viewing, and Recovering Faults.
2. From the Targets menu, select Middleware.
3. On the Middleware page, click the SOA Infrastructure target.
4. On the SOA Infrastructure target page, click Error Hospital.
5. In the Error Hospital tab, set the search criteria. For more information, see Table 7-9.
6. Click Search.
7. View the results:

   • To view the aggregate count of errors for each fault, see the Total Faults column in the results table.

   • To hide or unhide columns in the table, from the View menu, select Columns, then select the column name you want to hide or unhide.

   • To filter or perform a fine search for a particular column, enter a search keyword in the text-box placed above the column header. For more information, see Limiting Faults Searched and Retrieved from the SOA Infrastructure.

   • To group the faults by different categories, select the relevant category. For more information, see Customizing the Error Hospital Report.

   • To recover the faults in bulk, click Bulk Recover. For more information, see Performing Bulk Recovery from the Error Hospital Tab.

1. Create an error report. See Generating an Error Hospital Report.
2. In the Error Hospital page, select the fault attribute by which data is aggregated. To do so, from the Group By menu select one of the following fault attributes. By default, the faults are aggregated by the Fault Name. However, you can select any of the following options:

   • Fault Code: Aggregates the fault code.

   • Fault Name: Aggregates the fault name. This aggregation option is selected by default.

   • Fault Type: Aggregates the fault type:

     • System: Network errors or other types of errors such as a database server or a web service being unreachable.

     • Business: Application-specific faults that are generated when there is a problem with the information being processed (for example, a social security number is not found in the database.

     • OWSM: Errors on Oracle Web Service Manager (OWSM) policies attached to SOA composite applications, service components, or binding components. Policies apply security to the delivery of messages.

   • JNDI Name: Aggregates the JNDI name (for example, eis/FileAdapter).

   • Composite: Aggregate faults by the SOA composite application name.

   • Fault Owner: Aggregate faults by the name of the service component, service binding component, or reference binding component that handled the fault. In some cases, this can be both the fault owner and fault location.

   • Fault Owner Type: Aggregates the type of component, service, or reference that handled the fault (for example, if a BPEL process service component owns the fault, BPEL is displayed).

   • Partition: Aggregates the partition of the SOA composite application in which the fault occurred.

   • HTTP Host: Aggregates the HTTP host on which the fault occurred.