B Troubleshooting Oracle SOA Suite and Oracle BPM Suite

B.1.1 Log Files and Thread Dumps from All Managed Servers

Table B-1 describes the log files to view and thread dumps to obtain.

$DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log

kill -3 managed_server_process_ID

$DOMAIN_HOME/servers/server_name/logs/owsm/msglogging/diagnostic.log

B.2.1 Executing the Parallel Purge Script

This section addresses situations in which the Oracle SOA Suite database has not been managed and space must be reclaimed. This section also describes how to identify the appropriate parameter settings for normal, daily execution.

When trying to reclaim space, the purge scripts must be executed frequently to delete substantially more SOA composite applications than are being created. This has implications for host and database resources that must be monitored to avoid lengthy wait queues.

There must be a clear goal on the number of SOA composite applications to delete. Otherwise, the purge script can be configured too aggressively. The following aspects are described:

• Specifying the Degree of Parallel Value

• Parsing SOA Composite Applications to Delete

• Using Parallel Query Slaves

• Debugging and Tracing Purging Operations

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

ALTER SESSION FORCE PARALLEL DML STATEMENT

INSERT /*+ PARALLEL(TBL_INS,2) */ INTO TBL_INS
SELECT /*+ PARALLEL(TBL_SEL,4) */ * FROM TBL_SEL;

B.2.2 Oracle SOA Suite Table Partitioning

Oracle table partitioning addresses many concerns about the maintenance of large tables by decomposing them into smaller, more manageable segments called partitions. The SOA schema has been instrumented with partition keys to take advantage of the Oracle database range partitioning.

For more information about partitioning, see Section 10.4, "Partitioning Component Tables."

B.2.3 Reducing Audit Levels

This section provides tuning information about how to reduce the audit level for SOA composite applications and the data written to the Oracle SOA Suite schema.

B.3.1 Resolving Connection Timeouts

You can receive a connection timeout error under circumstances such as the following:

• You run a SOA composite application with a large payload that takes more than 30 seconds to process.

• You are invoking a stress test using a large payload from the Test Web Service page of Oracle Enterprise Manager Fusion Middleware Control.

• You are passing a large number of message files (one million) into a composite with a file adapter service.

• You are retrieving instance and fault count metrics in Oracle Enterprise Manager Fusion Middleware Control.

To avoid receiving timeout errors, increase the transaction timeout property as follows:

1. Log into Oracle WebLogic Administration Console.

2. Click JTA.

3. Change the value of Timeout Seconds (the default is 30).

4. Click Save.

5. Restart Oracle WebLogic Server.

B.3.2 Increasing Database Connection Values

You can receive the error message shown in Example B-1 because of slow connections to the database.

Exception [TOPLINK-4002] (Oracle TopLink - 11g Release 1 (11.1.1.1.0) (Build 
090304)): oracle.toplink.exceptions.DatabaseException 
Internal Exception: java.sql.SQLException: Internal error: Cannot obtain 
XAConnection weblogic.common.resourcepool.ResourceDeadException: Pool 
SOADataSource has been disabled because of hanging connection tests, cannot 
allocate resources to applications.

If this occurs, perform the following steps:

1. Open the DOMAIN_HOME\bin\setSOADomainEnv.cmd file.

2. Uncomment the lines shown in bold.

   # 8331492: Value of weblogic.resourcepool.max_test_wait_secs is 10 
   # seconds. It can be increased by uncommenting line below if your database 
   # connections are slow. See SOA documentation for more details. 
    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
   -Dweblogic.resourcepool.max_test_wait_secs=30" 
   export EXTRA_JAVA_PROPERTIES 
   

3. Save your changes and restart the managed Oracle WebLogic Server.

B.3.3 Updating the EJB Transaction Timeout Value in the Deployment Archive After SOA Infrastructure Failure

Updating the transaction timeout value for the FacadeFinderBean property in Oracle WebLogic Server Administration Console under Deployments > expanded SOA Infrastructure Application > FacadeFinderBean > Configuration tab can result in the error shown in Example B-2 after restarting the SOA Infrastructure:

java.lang.IllegalArgumentException: Cannot convert value of type [$Proxy223
implementing
 oracle.bpel.services.workflow.verification.IVerificationService,org.springframe
work.aop.SpringProxy,org.springframework.aop.framework.Advised]
to required type
[oracle.bpel.services.workflow.verification.IVerificationService] for
property 'verificationService': no matching editors or conversion strategy found
Message icon - Warning Errors were encountered while performing this operation.

The SOA Infrastructure status is also displayed as failed.

This error is not specific to FacadeFinderBean; it also applies to any EJB that is part of the SOA Infrastructure application.

To resolve this error, you must manually modify the transaction timeout setting in your deployment archive.

To update the transaction timeout setting:

1. Open the fabric-ejb.jar file in your deployment archive.

2. Increase the transaction timeout value in the META-INF/weblogic-ejb-jar.xml file to a larger value.

3. Rejar the file.

4. Restart the managed server that includes the SOA Infrastructure by following the instructions in Section 3.2, "Stopping and Starting the Managed Server and SOA Infrastructure."

   Note:

   This issue may also occur while updating any EJBs deployed as part of the SOA Infrastructure application. If this issue occurs, you must update the corresponding contained JAR file for those EJBs in a similar fashion.

B.3.4 Long Running, Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time

You may have long running, synchronous calls to remote web services that end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application may return with an error. The server logs show JTA transaction timeouts, which can cause this behavior.

To check the JTA transaction timeout in the Oracle WebLogic Server Administration Console.

1. Log in to Oracle WebLogic Server Administration Console.

2. In the Domain Structure, select Services > JTA to check the timeout value.

   If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then you may need to revisit the design approach. It may be best for the external web service to be invoked as an asynchronous transaction.

   Increasing the JTA for supporting long running, synchronous transactions is simply an interim mechanism. For information on changing the JTA transaction timeout setting, see Section B.3.1, "Resolving Connection Timeouts."

3. If synchronous client invocations take a long time, check for any performance issues with the system and try to resolve them.

4. Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.

   For information on viewing and changing the syncMaxWaitTime property, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

5. To view and change the BPEL EJB transaction timeout settings, perform the following steps:

   1. Log in to Oracle WebLogic Server Administration Console.

   2. In the Domain Structure, click Deployments.

   3. Expand soa-infra > EJBs.

   4. Update the following EJBs:

      - BPELActivityManagerBean

      - BPELDeliveryBean

      - BPELDispatcherBean

      - BPELEngineBean

      - BPELFinderBean

      - BPELInstanceManagerBean

      - BPELProcessManagerBean

      - BPELSensorValuesBean

      - BPELServerManagerBean

   5. Click Save.

   6. Restart Oracle WebLogic Server.

6. For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust as needed.

B.4.1 Oracle SOA Suite Runtime Failure with a "Cannot read WSDL" Error

The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.

oracle.fabric.common.FabricException: Cannot read WSDL

To diagnose this issue:

1. Ensure that the endpoint for the reference is up and running.

   The WSDL or endpoint is stated in the error. The service can then be looked up from Oracle Enterprise Manager Fusion Middleware Control to check if it is active. Once the service is active and the endpoint is reachable, search for the SOA instance and retry it through the Recovery tab for the BPEL process service engine in Oracle Enterprise Manager Fusion Middleware Control.

2. For extensions/customizations, ensure that the correct URL is updated in the deployment configuration plan.

   For information about deployment plans, including examples of using sca_extractPlan to extract plans, see the "Customizing Your Application for the Target Environment Before Deployment" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

B.4.2 Automatic Recovery of BPEL Instances is Not Recovering A Specific Instance

BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke/callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.

To resolve this issue:

• Set the maximum number of messages to automatically recover.

  By default, the automatic recovery feature of Oracle BPEL Process Manager processes 50 messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.

  1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

  2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties > RecoveryConfig.

  3. Expand both RecurringScheduleConfig > maxMessageRaiseSize and StartupScheduleConfig > maxMessageRaiseSize.

     The default value is 50 for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of 0 causes no messages to be selected from the database (effectively disabling recovery). To recover more than 50 messages, set the property value to that value. Use this property to limit the impact of recovery on the server.

• Set the maximum number of automatic recovery attempts on a given message.

  You can also configure the number of automatic recovery attempts to submit in the same recoverable instance. The value you provide specifies the maximum number of times that invoke and callback messages are recovered. If the value is 0 (the default value), it recovers all messages. Once the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.

  To configure automatic recovery attempts for invoke and callback messages in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

  2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties.

  3. Select MaxRecoverAttempt, and enter a value in the Value field.

  4. Click Apply.

     It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures has been fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.

     For more information, see Section 13.2, "Configuring Automatic Recovery for Oracle BPEL Process Manager."

B.4.3 Some Composites Are Retried Multiple Times on Failure

When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.

The property GlobalTxMaxRetry (default value is 3) specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

To rectify this situation, customize the composite by specifically marking the nonidempotent activities with idempotentset set to false in the partner link settings section of the composite.xml file to prevent retries.

<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>

You cannot set the idempotent property in Oracle Enterprise Manager Fusion Middleware Control.

You can also set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

To set GlobalTxMaxRetry:

1. Right-click soa-infra (SOA_cluster_name).

2. Select SOA Administration > Common Properties.

3. Click More SOA Infra Advanced Configuration Properties.

4. Click GlobalTxMaxRetry.

5. In the Value field, enter an appropriate value.

6. Click Apply.

For more information about the idempotent property, see the "BPEL Properties Set Inside a Composite" section of Oracle Fusion Middleware Performance and Tuning Guide and the "Managing Idempotence at the Partner Link Operation Level" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

B.4.4 Application Transaction Does Not Complete and the Underlying Composite Is Stuck in a Running State

An application transaction is not completing. For example, a purchase order status may remain processing. Checking the composite instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.

Faults may occur for various reasons:

• A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).

• A BPEL activity invoked an external web service that was unavailable.

• A BPEL activity has already been terminated by the administrator using Oracle Enterprise Manager Fusion Middleware Control.

• A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ/JMS queue.

• A BPEL activity invoked an asynchronous ADF service, but because Oracle SOA Suite was unavailable, the callback message did not arrive.

• A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).

• A network error occurred.

To diagnose this issue:

1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

2. In the navigator pane, go to domain_name > SOA.

3. Click soa-infra (SOA_cluster_name).

4. Click the Instances tab.

5. Search for the composite instance, and click the instance ID.

   The Flow Trace page appears.

   If the instance is not visible (and the Audit Level is not set to Off on the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of Oracle SOA Suite.

   If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.

   If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.

6. In the Trace section, click the BPEL process.

7. Expand the BPEL audit trail to see the exact point at which the service faulted.

   This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.

8. Click the View Raw XML link.

   The same information is also available through this link, where you can see the error. For example:

   ...
   <message>Faulted while invoking operation "modifyUserRoles" on provider
    "UserService". </message><details>
   ...
   <tns:message>JBO-27023: Failed to validate all rows in a
    transaction.</tns:message>
   <tns:severity>SEVERITY_ERROR</tns:severity>
   ...
   <message>The transaction was rolled back. The work performed for bpel instance
    "451042" was rolled back to the previous dehydration point, but the audit
    trail has been saved.
   You can recover the instance from the recovery console by resubmitting the
    callback message or activity for execution.</message
   

   Since the instance was rolled back to its previous dehydration point, the status remains as Running.

9. In the Audit Trail and Faults tabs, make a note of the following:

   • Composite name (for example, POComposite)

   • Component (for example, UpdateGuid BPEL process)

   • BPEL instance ID (for example, bpel:451042)

   This is all used in the recovery of the instance, if it is recoverable. The Audit Trail window may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.

10. To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

11. Click the Recovery tab.

12. From the Type list, select Activity.

13. Specify the composite and component names captured in step 9, and click Search.

14. Find the specific BPEL instance ID. You can recover faults marked as Recoverable.

15. Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.

    Note:

    You can also search for recoverable messages from the Faults and Rejected Messages tab of the SOA Infrastructure by clicking the message and selecting the appropriate action from the Recovery Actions list.

16. If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, you may need to provide diagnostic information to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.

17. If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, you can view the server.log and server-diagnostic.log files to see the logging of the message metadata logged by the JRF web services infrastructure.

    In addition, the ADF diagnostic logs are also available to debug, if needed.

    1. Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite application that invoked it.

    2. When viewing the log in Oracle Enterprise Manager Fusion Middleware Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic domain) to view messages across the domain.

    3. In the Selected Targets section of the Log Messages page for the Oracle WebLogic Server domain, ensure that the search includes the ECID field with the value noted in step a.

    4. Search and view log records for the execution content ID (ECID) and note any issues.

       For a specific ECID, you find several root instances (top level clients). You must drill down to the appropriate instance to find a specific fault.

    5. Observe if the components completed successfully or completed with an error. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

See the following chapters for the various ways to recover transactions and messages:

• Chapter 8, " Managing SOA Composite Application Instances" to recover from the SOA composite application page in Oracle Enterprise Manager Fusion Middleware Control.

• Chapter 15, "Managing BPEL Process Service Components and Engines" to recover from the BPEL process service component and BPEL process service engine message recovery pages.

B.5.1 Task Assignment/Routing/Escalation Issues

Table B-2 describes symptoms, possible causes, and possible solutions for task assignment/routing/escalation issues.

Ruleset returned lists with
different list builder

Ruleset returned lists with different list builder

B.5.2 Task Action Issues

Table B-3 describes symptoms, possible causes, and possible solutions for task action issues.

B.5.3 Notification Issues

Table B-4 describes symptoms, possible causes, and possible solutions for notification issues.

[2012-07-05T09:13:05.246-04:
00] [soa_server1] [ERROR] []
[oracle.soa.services.workflow
.common] [tid:
[ACTIVE].ExecuteThread: '6'
for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
<anonymous>] [ecid:
dc2ff0568380b04f:23824bc7:13
8570a4e67:-8000-000000000000
1716,0] [APP: soa-infra] <.>
[[ORABPEL-0
at
oracle.bpel.services.workflo
w.task.notification.TaskNoti
fications.notifyForTask(Task
Notifications.java:481)
. . .
. . .

<%string('À titre de responsable fonctionnel, une demande d'approbation 
. . .

<%string('d'd')%>

<%string('d''d')%>

B.5.4 Task View Issues

Table B-5 describes symptoms, possible causes, and possible solutions for task view issues.

B.5.5 Task Attribute Mapping Issues

Table B-6 describes symptoms, possible causes, and possible solutions for task attribute mapping issues.

B.5.6 Task Report Issues

Table B-7 describes symptoms, possible causes, and possible solutions for task report issues.

Null Pointer Exception when
running Task Productivity
Report

B.5.7 Task History Issues

Table B-8 describes symptoms, possible causes, and possible solutions for task history issues.

<Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService>
 <BEA-000000><ADFc:
 /META-INF/adfc-config.xml: > 
  <Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService><
ADFC-52024> <ADFc: Duplicate
 managed bean definition for
 'aleCompBindings' detected.> 

<Error> <Deployer>
 <BEA-149265> <Failure
 occurred in the execution of
 deployment request with ID
 '1297964056778' for task
 '3'. Error is:
 'weblogic.management.Deployme
ntException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.Deploymen
tException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].

java.lang.IllegalStateExcepti
on: Attempt to validate an
already invalid RegionSite: 

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e
3444eb1b:-8000-0000000000002f
0a,0] [APP: FederatedApp_
application1] unable to
 dispatch JSP page: The
 following exception
 occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext
        at
javax.faces.webapp.UIComponen
tClassicTagBase.getFacesConte
xt(UIComponentClassicTagBase.
java:2122) 

http://server:port/FederatedApp
/test.jspx

http://server:port/FederatedApp/fac
es/test.jspx

B.5.8 Task Form/Action Issues

Table B-9 describes symptoms, possible causes, and possible solutions for task form/ action issues.

Caused By:
 java.lang.ClassNotFoundException:
oracle.bpel.services.datacontrol.ty
pes.Number 

<library-ref>
 <library-name>oracle.soa.worklist.w
 ebapp</library-name>
 <specification-version>11.1.1</spec
 ification-version>
</library-ref> 

B.5.9 Task Comments/Attachment Issues

Table B-10 describes symptoms, possible causes, and possible solutions for task comments/attachment issues.

<context-param>
  <!-- Maximum memory per request
(in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_MEMORY</param-name>
  <!-- Use 500K -->
  <param-value>512000</param-value>
</context-param>
<context-param>
  <!-- Maximum disk space per
 request (in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_DISK_SPACE</param-name>
  <!-- Use 5,000K -->
 <param-value>5120000</param-value>
</context-param>
<context-param>
  <!-- directory to store temporary
 files -->
<param-name>oracle.adf.view.faces.U
PLOAD_TEMP_DIR</param-name>
  <!-- Use an ADFUploads
 subdirectory of /tmp -->
<param-value>/tmp/ADFUploads/</para
m-value>
</context-param>

B.5.10 Design Time at Runtime Issues

Table B-11 describes symptoms, possible causes, and possible solutions for design time at runtime issues. Two design time at runtime tools are available for use:

• Oracle SOA Composer

• Task Configuration tab of Oracle BPM Worklist

B.5.11 Human Workflow API (Including SOAP/EJB) Usage Issues

Table B-12 describes symptoms, possible causes, and possible solutions for human workflow API (including SOAP/EJB) usage issues.

B.5.12 Oracle JDeveloper Data Control / Form Generation Issues

Table B-13 describes symptoms, possible causes, and possible solutions for Oracle JDeveloper data control/form generation issues.

<?xml version='1.0'
 encoding='UTF-8'?><jsp:root
xmlns:jsp="http://java.sun.com/JSP/
Page" version="2.1"xmlns:f="http://java.sun.com/jsf/core"
xmlns:h="http://java.sun.com/jsf/html"
xmlns:af="http://xmlns.oracle.com/a
df/faces/rich">  <jsp:directive.page
contentType="text/html;charset=UTF-8"/>
  <f:view>
    <af:document id="d1">
      <af:form id="f1"></af:form>
    </af:document>
  </f:view>
</jsp:root>

B.5.13 Human Workflow Service/ System MBean Browser Issues

Table B-14 describes symptoms, possible causes, and possible solutions for human workflow service/System MBean Browser issues.

file:///home/wstallar/wfcustomizati
ons/

B.5.14 AMX Extension Issues

Table B-15 describes symptoms, possible causes, and possible solutions for AMX extension issues.

$FMW_HOME/SOA_HOME/soa/modules/ora
cle.soa.ext_11.1.1/classes

B.5.15 Oracle BPM Worklist/Task Region Issues

Table B-16 describes symptoms, possible causes, and possible solutions for Oracle BPM Worklist/task region issues.

<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService>
<BEA-000000><ADFc:
/META-INF/adfc-config.xml: >
<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService><ADFC-5
2024> <ADFc: Duplicate managed
bean definition for
'aleCompBindings' detected.> 

Duplicate default server in
client configuration.
Configuration needs to have only
one default server in client
configuration.
Specify one default server in
client configuration. 

The default server is not
specified 

Invalid display column.
The display column COLUMN NAME
is not a valid Task column.
Specify a valid column name.

java.lang.IllegalStateException:
Attempt to validate an already
invalid RegionSite: 

Caused by:
oracle.adf.controller.Controller
Exception: ADFC-02001: The ADF
Controller cannot find
'/WEB-INF/taskList-task-flow-def
inition.xml'

<parameter
id="taskTypesFilterList"
value="http://xmlns.oracle.com/H
elpDeskRequestSOAApp/HelpDeskReq
uestComposite/HelpDeskRequestHum
anTask,[
http://xmlns.oracle.com/Vacation
RequestApp/VacationRequest/Vacat
ionRequestTask]"/>

<parameter id=
"attributesFilterOperator"
value="and"/>
 <parameter id= "attributesFilterList"
 value="state=ASSIGNED"/>

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000002f0a,0]
 [APP: FederatedApp_
application1] unable to dispatch
 JSP page: The following
 exception occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext 
at
javax.faces.webapp.UIComponentCl
assicTagBase.getFacesContext(UIC
omponentClassicTagBase.java:2122)

http://server:port/FederatedApp/test.jspx

http://server:port/FederatedApp/
faces/test.jspx

[AdminServer] [TRACE] [] []
[tid: [ACTIVE].ExecuteThread:
'5' for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000001d39,0]
[SRC_CLASS:
oracle.bpel.services.workflow.cl
ient.config.ClientConfigurationU
til] [APP: FederatedApp_
application1] [SRC_METHOD:
getClientConfiguration]
WorkflowServiceClientContext:
Cannot find client
configuration file: wf_client_
config.xml 

<Error> <Deployer> <BEA-149265>
<Failure occurred in the
execution of deployment request
with ID '1297964056778' for
task '3'. Error is:
'weblogic.management.DeploymentE
xception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.DeploymentEx
ception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false]. 

<parameter id= "displayColumnsList"
value="assignees,creator,
assignedDate,state,
textAttribute1,textAttribute2"/>

<parameter
 id="taskTypesFilterList"
value="http://xmlns.oracle.com/Hel
pDeskRequestSOAApp/HelpDeskRequest
Composite/HelpDeskRequestHumanTask
"/>

B.5.16 Test-to-Production Issues

Table B-17 through Table B-20 describe symptoms, possible causes, and possible solutions for test-to-production issues.

<env:Envelope
xmlns:env="http://schemas.xmlsoap.org/soap
/envelope/">
 <env:Header/>
 <env:Body>
  <realmName
xmlns="http://xmlns.oracle.com/bpel/servi
ces/IdentityService">jazn.com</realmName>
 </env:Body>
</env:Envelope>

[java] Error in workflow service Web
service operation invocation. The
error is ORA-30501:Error in
authenticating user.
[java] Error in authenticating and
creating a workflow context for
user jazn.com/FMW_USERID. [java]
Verify that the user credentials
and identity service configurations
are correct.

[java] Caused by:
java.sql.SQLIntegrityConstrain
tViolationException: ORA-02291:
integrity constraint (UAT_
SOAINFRA.SYS_C0018364) violated -
parent key not found* * 

[java]  Error encountered during
migration. [java] Exception in thread
"main"[java] 
UserConfigDataMigrationException:[java]
faultString:Invalid parameters for RULE.
[java] Invalid user and group: both
parameters can not have null
values.[java] To migrate User Rules,
provide the 'user' parameter only.
[java]  To migrate Group Rules, provide
the 'group' parameter only.[java]
oracle.bpel.services.workflow.util.
tools.wfUserConfigDataMigrator.
UserConfigDataMigrationE xception [java]
at oracle.bpel.services.workflow.
util.tools.wfUserConfigDataMigrator.
implhwfMigrator.parseParametersNode

[java]  Error encountered during
migration.[java] Exception in
thread "main"[java] 
UserConfigDataMigrationExcepti
on: [java]  faultString:
Invalid parameters for RULE.
[java]  Invalid user and group: both
 parameters can not have values.
[java]  To migrate User Rules,
 provide the 'user'parameter only. 
[java]  To migrate Group Rules,
 provide the 'group' parameter only.
[java]  oracle.bpel.services. workflow.util.tools.wfUserConfigData
Migrator.UserConfigDataMigrationE
xception 

http://host:port/integration/worklistapp/

http://host:port/integration/services
/TaskQueryService/TaskQueryService

http://host:port/integration/services/IdentityService/configuration

http://host:port/integration/services/IdentityService/identity

http://host:port/integration/services/RuntimeConfigService/RuntimeConfigService

For more information about test-to-production issues, see Section 23.6, "Moving Human Workflow Data from a Test to a Production Environment."

B.5.17 Identity Service Issues

Table B-21 and Table B-22 describe symptoms, possible causes, and possible solutions for identity service issues.

Token Decryption Failed

UserDN 1 : cn=users1,dc=us,dc=oracle,dc=com
UserDN 2 : cn=users2,dc=us,dc=oracle,dc=com
UserDN 3 : cn=users3,dc=us,dc=oracle,dc=com

dc=us,dc=oracle,dc=com

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
 <property name="idstore.config.provider" 
value="oracle.security.jps.wls.internal.idstore.WlsLd
apIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
 <extendedProperty>
   <name>user.search.bases</name>
   <values>
      <value>cn=users1,dc=us,dc=oracle,dc=com</value>
      <value>cn=users2,dc=us,dc=oracle,dc=com</value>
   </values>
 </extendedProperty>
</serviceInstance>

<serviceInstance name="idstore.ldap"
  provider="idstore.ldap.provider">
   ............................
   <property name="virtualize" value="true"/>
   ..............................
</serviceInstance> 

Exception seen :
 Service"
Unknown macro: {0}
"in configuration" 
Unknown macro: {1}
" could not be
initialized. Error
in initializing
service
"Authentication"
in configuration
"myrealm".

No Role found
matching the
criteria

GroupDN 1 : cn=groups1,dc=us,dc=oracle,dc=com
GroupDN 2 : cn=groups2,dc=us,dc=oracle,dc=com
GroupDN 3 : cn=groups3,dc=us,dc=oracle,dc=com

dc=us,dc=oracle,dc=com

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
   <property name="idstore.config.provider"
value="oracle.security.jps.wls.internal.idstore.WlsL
dapIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
  extendedProperty>
  <name>group.search.bases</name>
  <values>
    <value>cn=groups1,dc=us,dc=oracle,dc=com</value>
    <value>cn=groups2,dc=us,dc=oracle,dc=com</value>
  </values>
 </extendedProperty>
</serviceInstance>

B.6.1 Tuning EDN Event Bus and Delivery

You can customize the following EDN properties in the System MBean Browser:

1. Right-click soa-infra.

2. Select SOA Infrastructure > Administration > System MBean Browser.

3. Expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server-x > EDNConfig > edn.

4. Double-click an attribute.

   Properties	Type	Details
   NumberOfThreads	int	Defaults to 3 for the EDN-DB implementation or 2 for the EDN-JMS implementation. Setting this property to 0 is effectively the same as 2 for EDN-DB or 1 for EDN-JMS. For more information about EDN-DB and EDN-JMS, see Section 35.1, "Introduction to the Event Delivery Network."
   Paused	boolean	Values are true or false. If true, the EDN listener threads are decreased to 0, which effectively stops the delivery of events. This works for EDN-DB and EDN-JMS.
   NumberOfRetrys	int	The maximum number of EDN deliveries and redeliveries before moving the event to the error hospital. This parameter works only for one-and-only-one delivery, and not for guaranteed delivery. The maximum allowed value is 5. If set to less than 5, then it is effectively the same as 5. If NumberOfRetrys is set to the following values: >= 2, then EDN delivers the event once, and may try redeliveries up to (NumberOfRetrys -1) times in case of XA failures. = 1, then EDN delivers the event only once. There is no retry with XA failures. = 0, then this is the same as = 1 (that is, EDN delivers an event only once). There is no retry with XA failures. > 5, then it is effectively the same as NumberOfRetrys = 5.

   
   

5. Enter a value, and click Apply.

B.6.2 Rolled Back One-and-Only-One Event Delivery Messages are Displayed in the Log Files

If the Java debug level is set to TRACE:16 (FINER) or a lower value, you may see log messages such as the following:

Began XA for OAOO
Rolled back XA for OAOO

These are normal messages of one-and-only-one (OAOO) event delivery when there are no events waiting to be delivered. These are not error conditions. You can turn off these messages by setting the Java logging level to TRACE:1 (FINE) or a higher value.

B.6.3 Events Are Consumed by Multiple Revisions of the Same Composites

By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when you have multiple active versions of the composite that may be a result of a patch failure.

Perform the following steps to determine whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:

1. Go to Oracle Enterprise Manager Fusion Middleware Control.

2. Click SOA > soa-infra.

   In the Deployed Composites section of the Dashboard tab, you see a list of deployed composite names, revisions, and modes (for example, active).

3. Identify composites with the same name and with an active mode, but with different revisions.

4. Click the composite revision that should not be active.

5. Click the Retire button.

B.6.4 Business Event Is Picked Up Twice (Or More) By SOA Server

Business events may be raised from J2EE applications and picked up more than once. Expected behavior is that they must be picked up only once.

If a patch fails, it may be possible for two versions of a given composite to be active (that is, the older version has not been retired). When multiple versions of a composite are active, they all become subscribers and the event is picked up more than once. This should not happen under normal scenarios. Reapply the patch and ensure that the deployed composite only has one active version.

B.6.5 Some Messages Are Lost Between EDN and Composites or Composites Across Clusters

Messages primarily may get lost for the following reasons:

1. The EDN message was not delivered.

2. The target asynchronous service did not respond.

3. The message was roll backed (though it was not lost; you still see it in Oracle Enterprise Manager Fusion Middleware Control).

This issue typically requires you to identify where the messages are supposed to be and to diagnose the path.

• Check if the Oracle WSM Policy Manager security configurations allow the client to invoke the SOA composite.

• Verify that the transaction is not transient (nonpersistent).

  Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes impact the dehydration store database in different ways.

  • Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process you design in Oracle JDeveloper is an example of a transient process.

  • Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:

    • Receive activity

    • OnMessage branch of a pick activity

    • OnAlarm branch of a pick activity

    • Wait activity

    Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

• If a composite instance is not visible in Oracle Enterprise Manager Fusion Middleware Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off on the SOA Infrastructure Common Properties page. This can be checked in Oracle Enterprise Manager Fusion Middleware Control:

  1. Right-click soa-infra (SOA_cluster_name).

  2. Select SOA Administration > Common Properties.

• If the composite instance is available, check the oneWayDeliveryPolicy BPEL property value. You can check the value in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigation pane, expand soa-infra (SOA_cluster_name).

  2. Expand the partition, and select the composite.

     The Dashboard page for the composite is displayed.

  3. In the upper right corner, click the Show XML Definition icon.

     The contents of composite.xml for that composite are displayed.

  If this is set to async.cache, you may lose messages. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the composite.xml file. Therefore, this can be set for custom composites. If the value is not set in composite.xml, the value for oneWayDeliveryPolicy in the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control is used. The following values are possible:

  • async.persist: Messages are persisted in the database hash map.

  • async.cache: Messages are stored in memory.

  • sync: Direct invocation occurs on the same thread.

  For more information about these settings, see the "Deployment Descriptor Properties" appendix and the "Transaction and Fault Propagation Semantics in BPEL Processes" chapter of the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

  For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

  It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the composite instance flow shows the call to the target asynchronous service.

• Check if the instance has rolled back and the message is in recovery.

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

  2. Right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

  3. Click the Recovery tab.

  This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).

B.6.6 Checking Whether Bad Composites Exist in the SOA Domain that Slow Down Overall EDN Event Delivery

EDN internally uses two advanced queuing (AQ) queues in a serial processing fashion for all business events. One or more bad SOA composite applications that fail to consume certain types of events can slow down EDN delivery of all other event types and cause an event backlog in the AQ queues.

1. Check if there are repeated errors from bad SOA composite applications in the SOA server log with EDN logging enabled. Even if the errors are in completely separate SOA composite applications that are unrelated to the event types of interest, it can delay the overall delivery of events of interest to the target SOA composite applications.

2. Check the event backlog count as detailed in Step 4 (events are stuck in advanced queue). Obtain the counts by running the following SQL queries (substitute SOAINFRA-SCHEMA-USER with the schema owner).

   SELECT COUNT(*) FROM SOAINFRA-SCHEMA-USER.EDN_OAOO_DELIVERY_TABLE; 
   SELECT COUNT(*) FROM SOAINFRA-SCHEMA-USER.EDN_EVENT_QUEUE_TABLE;
   

   If bad composites are found in Step 1 and large numbers of backlogged events occur in Step 2, then proceed to 3.

3. Stop bad composites identified in Step 1. This can be achieved from Oracle Enterprise Manager Fusion Middleware Control or the WLST command line utility.

4. Temporarily increase the number of EDN threads to a larger number, (for example, from 3 to 5 or 7) to clear up the backlogged events faster.

   To change the number of EDN threads through the configuration of the EDNConfig MBean in Oracle Enterprise Manager Fusion Middleware Control:

   1. Go to the navigation pane.

   2. Right-click soa-infra.

   3. Select SOA Infrastructure > Administration > System MBean Browser.

   4. Expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server-x > EDNConfig > edn.

   5. Change the value for the NumberOfThreads attribute, and click Apply.

      If you do not care about backlog event processing, you can purge the backed up events directly in AQ. The purge script is as follows:

      DECLARE
        purge_options dbms_aqadm.aq$_purge_options_t;
      BEGIN
        purge_options.block := FALSE;
        DBMS_AQADM.PURGE_QUEUE_TABLE(
          queue_table     => '&edn_user..edn_event_queue_table',
          purge_condition => NULL,
          purge_options   => purge_options);
        DBMS_AQADM.PURGE_QUEUE_TABLE(
          queue_table     => '&edn_user..edn_oaoo_delivery_table',
          purge_condition => NULL,
          purge_options   => purge_options);
      END;
      /
      commit;
      

5. Periodically monitor the EDN backlog count (see Step 2) to confirm that it is being reduced. If the clear-up rate is very slow, you can further increase the EDN thread number (see Step 4).

6. Continue to monitor the EDN backlog count over a period. Also monitor the recent instances and faults in Oracle Enterprise Manager Fusion Middleware Control to see if composite instances are being created or new faults are showing up both across the board (that is, at SOA Infrastructure level) and for the composite of interest.

7. Once the backlog is cleared up, revert the number of EDN threads to the previous setting or an appropriate number.

8. Publish a business event of interest and verify that the target composite is triggered with expected instance creations.

B.7.1 Optimizing the Loading of Pages with Instance and Fault Metrics

Since production systems can include numerous composite instances and faults, there is a possibility of timeouts in the Oracle Enterprise Manager Fusion Middleware Control pages as information retrieval becomes relatively slow. To optimize the loading performance of Dashboard pages for the SOA Infrastructure, SOA composite applications, service components, and service engines; Deployed Composites page of the SOA Infrastructure and service engines; and partition home page, you can enable two property settings on the SOA Infrastructure Common Properties page that perform the following:

• Disable the loading of all metrics information upon page load. The instances and faults metrics can be obtained on demand from the server.

  Note:

  This setting disables fault, instance, and count metrics, and not DMS metrics.

• Restrict the retrieval of instances, faults, and counts to a specified time period (default selection).

To optimize the loading of pages with instance and fault metrics:

1. In the navigator, click soa-infra.

2. Note that values appear in the Running and Total fields in the Recent Composite Instances section and the Instances and Faulted Instances columns of the Deployed Composites section. When these values are large, it can take time to load this page and other pages with similar information.

   
   Description of the illustration soaadmin_fetch1.gif
   
   

3. From the SOA Infrastructure menu, select SOA Administration > Common Properties.

4. In the Data Display Options section, select the Disable fetching of instance and fault count metrics checkbox.

5. Click Apply.

6. Return to the Dashboard page of the SOA Infrastructure.

7. Note that the values that previously displayed have been replaced with links.

8. In the Deployed Composites section, click a link for a specific SOA composite application to display details about the number of instances. The values are calculated for the link you select.

   
   Description of the illustration soaadmin_fetch7.gif
   
   

9. Click OK.

10. Click Number of Instances.

    
    Description of the illustration soaadmin_fetch6.gif
    
    

    A message displays the number of running and total instances in the SOA Infrastructure.

11. Click OK.

12. Return to the SOA Infrastructure Common Properties page.

13. In the Data Display Options section, select the Restrict display of instances and faults to the last time_period checkbox if it is not already selected. By default, this checkbox is selected and the time period duration is set to 24 hours (one day).

14. Specify a time period during which to retrieve recent instances, faults, and count metrics for display (for example, 10 minutes).

15. Click Apply.

16. Return to the Dashboard page of the SOA Infrastructure.

17. Note that the following message is displayed in the upper right section of the page:

    Recent Instances and Faults for the last 10 minutes
    

18. Click Number of Instances.

    The value that you specified for the time period is displayed in the message title. The number of running and total messages for that time period is also displayed.

    
    Description of the illustration soaadmin_fetch8.gif
    
    

19. Click OK.

20. In the Deployed Composites section, click a link.

    
    Description of the illustration soaadmin_fetch2.gif
    
    

    The value that you specified for the time period is displayed in the message title. The number of running and total messages for the selected SOA composite application for that time period is also displayed.

    
    Description of the illustration soaadmin_fetch3.gif
    
    

For more information about setting these properties, see Section 3.1, "Configuring SOA Infrastructure Properties."

B.7.2 Resolving Message Failure Caused by Too Many Open Files

You can receive the following error at runtime or compilation time, depending on the number of JAR files being used, the use of file descriptors by JDK 6/JRE, or both.

Message send failed: Too many open files 

To resolve this error, increase the number of file descriptors to at least 4096.

1. Use the limit command (for the C shell) or the ulimit command (for the Bash shell) to identify the value for descriptors. A value of 1024 is typically too low, especially for JDK 6.

   % limit
   
   cputime      unlimited
   filesize     unlimited
   datasize     unlimited
   stacksize    10240 kbytes
   coredumpsize unlimited
   memoryuse    unlimited
   vmemoryuse   unlimited
   descriptors  1024
   memorylocked 500000 kbytes
   maxproc      46720
   

2. Log in as the root user on your operating system.

3. Edit the /etc/security/limits.conf file to increase the value for descriptors.

   For this example, the limits.conf file appears as follows after increasing the limit for all users to 4096:

   #<domain>      <type>  <item>         <value>
   #
   
   #*               soft    core            0
   #*               hard    rss             10000
   #@student        hard    nproc           20
   #@faculty        soft    nproc           20
   #@faculty        hard    nproc           50
   #ftp             hard    nproc           0
   #@student        -       maxlogins       4
   
   # End of file
   @svrgroup    soft    memlock         500000
   @svrgroup    hard    memlock         500000
   *           soft    nofile          4096
   *           hard    nofile          4096
   

4. Close your terminal and reopen for the change to take effect. A system restart is not required.

B.7.3 Resolving MaxMessageSizeExceededException Errors Caused By Large Payloads

If you provide a large payload (for example, 12 MB) to your deployed SOA composite application, then click View XML Document in the audit trail to view the payload, you can encounter MaxMessageSizeExceededException errors. This error can be resolved by setting the following JVM parameter.

1. Open the following file:

   • On UNIX operating systems, open $MIDDLEWARE_HOME/user_projects/domains/domain_name/bin/setDomainEnv.sh.

   • On Window operating systems, open MIDDLEWARE_HOME\user_projects\domains\domain_name\bin\setDomainEnv.bat.

2. Add the weblogic.MaxMessageSize property with the following value:

   EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
   -Dweblogic.MaxMessageSize=20000000"
   export EXTRA_JAVA_PROPERTIES 
   

3. Restart the server.

B.7.4 Extending Tablespaces to Avoid Problems at Runtime

If the database tablespace is not extended, runtime processing can be impacted. Messages are not processed or persisted, and exception errors similar to that shown in Example B-3 can appear in the log files. This is because Oracle BPEL Process Manager relies on the database to store instance data. If the database is not available, runtime processing is impacted.

INFO: MediatorServiceEngine returning after processing the request for 
operation = processResponse 

[EL Warning]: 2009.01.14 11:46:16.783--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 

Error Code: 1691 
Query: InsertObjectQuery(com.collaxa.cube.persistence.dto.AuditTrail@199b33d) 
[EL Warning]: 2009.01.14 11:46:16.782--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 
. . .
. . .

Ensure that you set a tablespace to automatically extend itself by a specified amount when it reaches its size limit. If you do not enable autoextend, ensure that you respond when alerted that the tablespace is reaching its critical or warning threshold size. You can respond to size alerts by manually increasing the tablespace size.

B.7.5 Resolving Database Growth Issues Caused by a High Volume of Transactions

If Oracle SOA Suite transactions fail and logs indicate that the database is running out of space, the dehydration store or MDS store may be running out of space due to a high volume of transactions. In the latter example, you see errors such as the following:

java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by
16 in tablespace FUSION_TS_TOOLS 

This indicates that the tablespace is full and the database cannot extend it.

Perform the following tasks.

1. Purge the dehydration store tables periodically, taking into account the appropriate record retention policies and ensuring that the applications have no dependencies on runtime data.

   The purge should be followed by commands to coalesce the space. For the purging strategy to work, it is important to understand how long to retain the data in the database. Factors that drive the retention policy include the following:

   • Legal requirements

   • Line of business requirements

   • Overall company policy on retention of data

   The longer the retention policy, the greater the volume of data that must be stored and, correspondingly, the higher the disk capacity requirements.

   For details on creating a purging strategy, see Chapter 9, "Developing a Database Growth Management Strategy" and Chapter 10, "Managing Database Growth."

2. Ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning before configuring your tables for partitioning.

   For the dehydration store, database partitioning using range partitioning and hash partitioning is an optimal solution. Partitioning by definition means storing data in multiple tables to reduce bigger data sets into smaller, more manageable data sets. Partitioning strategies play a large role in easing maintenance overheads (dropping and pruning the partition) and improving performance. Partitioning should at least be done for tables having high activity. This plays a large role in balancing disk I/O and preventing hot disks. One important requirement that you must meet before configuring your tables for partitioning is to ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning. If preproduction testing has indicated that the installation is large, Oracle expects that you have sized your environment (CPU, memory, and disk space) correctly to take advantage of the partitioning features.

3. Tune database parameters for memory, tablespace, and partitions to get maximum performance. For more information, see Section "Tuning Database Parameters" of the Oracle Fusion Middleware Performance and Tuning Guide.

4. For other tablespaces running out of space, use the following query to check for free tablespace:

   SELECT TOTAL.TABLESPACE_NAME "TABLESPACE NAME", FREE_SPACE, (TOTAL_SPACE-FREE_
   SPACE) USED_SPACE, TOTAL_SPACE, ROUND((FREE_SPACE*100/TOTAL_SPACE),2) "FREE %"
   FROM (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) FREE_SPACE FROM SYS.DBA_
   FREE_SPACE GROUP BY TABLESPACE_NAME) FREE,
   (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) TOTAL_SPACE FROM SYS.DBA_DATA_
   FILES GROUP BY TABLESPACE_NAME) TOTAL
   WHERE FREE.TABLESPACE_NAME = TOTAL.TABLESPACE_NAME AND TOTAL.TABLESPACE_NAME =
    '<TABLESPACENAME>'
   ORDER BY 5;
   

5. To increase tablespace settings, use the administrator account. For example:

   ALTER TABLESPACE TABLESPACE_NAME ADD DATAFILE 'DATAFILE_NAME' SIZE 500M
   AUTOEXTEND ON;
   

For more details, see Section B.7.2, "Resolving Message Failure Caused by Too Many Open Files."

B.7.6 Observing Slow Application Performance Such as Longer Time to Serve Pages or Finish Transactions

You can observe slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.

There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking in the database. Thread contention can also be a reason for poor performance.

To diagnosis this issue:

1. Check the CPU utilization to see if it is saturated due to a heavy load or too many processes in relation to CPU capacity.

   If CPU utilization is at 100% during normal load hours (the target should be 70-80%), you have no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.

2. Check applications using Oracle Enterprise Manager Fusion Middleware Control to report on performance. Check the performance of services and the invoke/response times in the BPEL process audit trail.

3. To maximize performance, it is recommended that you not set the logging level higher than the default INFO level.

   For debugging purposes, you must set the logging level to the FINEST level. However, once issues are resolved, reset the logging level to the default level for best performance. It is also recommended that you set Audit Level to Production on the SOA Infrastructure Common Properties page. This can be set in Oracle Enterprise Manager Fusion Middleware Control as follows:

   1. Right-click soa-infra (SOA_cluster_name).

   2. Select SOA Administration > Common Properties.

   3. Set Audit Level to Production.

4. Purge periodically based on retention requirements to maintain any service level agreements (SLAs).

   To identify the tables where data growth may lead to performance issues, see Section "Identifying Tables Impacted By Instance Data Growth" of the Oracle Fusion Middleware Performance and Tuning Guide.

   For information about using the purge scripts, see Chapter 10, "Managing Database Growth."

5. Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.

6. Optimize the JVM to avoid full garbage collection or out-of-memory errors.

   Frequent garbage collection can be either due to higher memory usage or memory leaks.

   1. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM to avoid operating system level paging activity.

   2. Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues. For more information about thread dumps, see Chapter 12, "Diagnosing Problems with SOA Composite Applications."

7. Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.

8. Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.

9. Run database AWR snapshots to identify causes of database performance issues.

10. Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.

    Here are some common tuning recommendations. For more information, see the "Tuning Database Parameters" section in the Oracle Fusion Middleware Performance and Tuning Guide.

    • Put indexes and tables in as physically separate disk areas, if possible.

    • Never put rollback segments with data or index segments.

    • Separate highly active tables and indexes into their own tablespaces.

    • Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.

    • Have processes in place to generate database table statistics at regular intervals.

11. Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control HWM contention and ensure consistent performance of producing and consuming messages from AQ.

    The EDN AQ names to be aware of are EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE.

12. Tune the BPEL process and EDN thread counts to ensure optimal settings (for example, the Dispatcher Invoke Threads and Dispatcher Engine Threads properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control). If the thread configuration is too high, the servers run out of memory. If they are too low, the messages start backing up.

13. Tune the BPEL process properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control), if they are not needed.

14. In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.

For more information about performance tuning the various components, see the "Top Performance Areas" chapter and the "SOA Suite Components" part in the Oracle Fusion Middleware Performance and Tuning Guide.

For information about setting properties on the BPEL Service Engine Properties page, see Chapter 13, "Configuring BPEL Process Service Components and Engines."

B.7.7 Observing Incoming Message Rates Exceeding Outgoing Message Rates

When the incoming rate of messages exceeds the processing rate of messages (that is, the outgoing rate), you can receive the warning message shown in Example B-4.

[2012-04-09T16:49:40.342-04:00] [Soa_server1] [WARNING] []
 [oracle.soa.bpel.engine.dispatch] [tid:
 weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@e3d0e3d
] [userId: <anonymous>] [ecid: 0000JSnNKVTBh4Z5ln1FiZ1Fdnmd001gWL,0] [APP:
soa-infra] BPEL service engine is overloaded, the inflow is higher than the
outflow, there are "100000" invoke messages pending in the dispatcher in-memory
cache, the configured threshold is "102",the new incoming invoke messages will not
be dispatched or processed immediately, they are going to be persisted in the
dehydration store and can be recovered later using recovery.

To resolve this issue, either increase the processing rate or lower (that is, throttle down) the incoming rate. The relatively low processing rate may be due to any or all of the following issues:

• Not enough invoke threads

• Not enough JVM capacity

• The external partners that the BPEL process is calling cannot scale, which causes the messages to back up in the BPEL process.

To identify the exact causes and resolve them, perform the following steps:

1. Check the oneWayDeliveryPolicy property value in the composite.xml file. Is it set to async.persist, async.cache, or sync? For information about this property, see Appendix "Deployment Descriptor Properties" of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

2. Ensure that the outgoing invocations from the BPEL process to external services are not backed up due to low scalability of the external services. If the bottleneck is in an external service, any tuning you perform in Oracle SOA Suite (that is, Steps 3 and 4) does not help. You must first resolve the bottleneck.

3. Monitor your JVM garbage collection (GC). If the GC frequency appears higher than normal and the JVM appears to be near capacity, increase the JVM heap size. If the JVM does not have enough capacity, any tuning you perform in Step 4 makes matters worse.

4. Increase the values for the dispatcher invoke threads and dispatcher engine threads on the BPEL Service Engine Properties page so that the server has more threads to process the incoming messages. For more information, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

B.8.1 Best Practices for Starting and Stopping a Managed Server

As a best practice, it is always recommended that you start and stop a managed server through one, but not both, of the following methods. Do not mix these methods, such as starting the managed server from the command line and stopping it from Oracle Enterprise Manager Fusion Middleware Control, or vice versa.

• Oracle Enterprise Manager Fusion Middleware Control

  With this method, the node manager must be up and running. The node manager tracks all managed server startups and shutdowns performed from Oracle Enterprise Manager Fusion Middleware Control. With this method, the server state is not an issue.

• Command line

  With this method, the node manager does not track the server state. Therefore, if you start the server from the command line and shut it down from Oracle Enterprise Manager Fusion Middleware Control, the Oracle WebLogic Administration Server accesses the node manager to determine its status, which returns a state of unknown.

Perform the following steps to stop and start the server from Oracle Enterprise Manager Fusion Middleware Control.

1. Expand the WebLogic domain.

2. Select the managed server (for example, named soa_server1).

3. Select Control > Shut Down.

4. Select Control > Start Up.

For information on starting and stopping managed servers from the command line, see Oracle Fusion Middleware Installation Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

B.8.2 Diagnosing SOA Server Startup Problems

If the SOA server (soa_infra) does not start, there can be various reasons for server startup issues. A discovery-based approach to finding the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following.

To diagnose server startup issues:

1. Check if the database is not available or there are not enough connections available. Some failures with Oracle SOA Suite runtime can result from database outage/connectivity issues. Perform the following steps:

   1. Log in to Oracle WebLogic Server Administration Console.

   2. In the Domain Structure, view the status by selecting Services > Data Sources > SOADatasource > Monitoring > Testing to test the data source. You can also check this from a SQL prompt.

2. Check the list of ports used for port conflicts.

3. Check if the MDS repository cannot load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).

4. Check the Oracle Coherence configuration if using an Oracle SOA Suite cluster (use of unicast versus multicast).

   This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Oracle Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:

   [soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] 
   [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] 
   [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 
   <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
    (self-tuning)', member=n/a): 
   Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout
    during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
   

   In this example, Oracle Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:

   Error creating bean with name 'SensorManager' defined in ServletContext
   resource
   

   The root cause of this is still primarily related to Oracle Coherence configuration.

5. Check if both the administration server and managed server ports are open and accessible.

6. Check if managed server startup failed because the administration server is not reachable.

7. Check network issues (for example, IP routing filtering/rules that may be causing issues).

8. Check Oracle WebLogic Server LDAP security corruption.

   The managed server may report that policies for the application System MBeans Browser already exist or do not exist.

For more information, see the "Setting the Frontend URL for the Administration Console and Setting Redirection Preferences" section and the "Setting the Frontend HTTP Host and Port" section in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

B.8.3 Specifying the Proxy Server

To use system properties to specify the proxy server, write your client application in the standard way, and then specify Java system properties when you execute the client application. Example B-5 provides details.

setenv PROXY_SETTINGS "-DproxySet=true
-Dhttp.proxyHost=www-myproxy.us.mycompany.com -Dhttp.proxyPort=80
 -Dhttp.nonProxyHosts=localhost|*.us.mycompany.com
  |0:0:0:0:0:0:0:1|fe80:0:0:0:250:56ff:fe31"

B.8.4 Flow Diagram Does Not Display The First Time on Some Lower End Hosts

The flow diagram for an instance ID of a deployed SOA composite application in Oracle Enterprise Manager Fusion Middleware Control may not display the first time on some lower end hosts. Instead, you receive a failed to load resource message.

As a workaround, close the flow trace page and click the instance ID to return to the flow trace page.

B.8.5 Accessing Oracle Enterprise Manager Fusion Middleware Control on Dual Stack Hosts that Support IPv4 and IPv6

If you run Oracle SOA Suite on a dual stack host that supports both IPv4 and IPv6, you must update the etc/hosts file as shown in Table B-23 for IPv4 clients to access IPv6 URLs in Oracle Enterprise Manager Fusion Middleware Control.

xx.xxx.xxx.xxx myhost10-ipv6 

2001:0db8:db0a::0:1  myhost10-ipv6 myhost10-ipv6.us.example.com

B.9.1 Limitation on Using the Safari Browser to View WSDL File Content

If you are using the Safari browser, note the following limitation and workaround for viewing WSDL file contents in Oracle Enterprise Manager Fusion Middleware Control. Note also that Mozilla Firefox works correctly and does not require this workaround.

1. Go to the home page for a SOA composite application.

2. Click the Show WSDL and endpoint URI link at the top of the page.

3. Click the WSDL link that is displayed.

   This opens a blank page that does not display the contents of the selected WSDL.

   As a workaround, perform the following additional steps.

4. In the upper right corner of this page, click the Display a menu for the current page icon.

5. Select View Source from the menu that is displayed.

   This displays the contents of the selected WSDL in another page.