43 Managing Large Documents and Large Numbers of Instances

43.1.1.1.1 SOAP Inline

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 43-1 provides details.

43.1.1.1.2 SOAP MTOM

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed as a Message Transmission Optimization Mechanism (MTOM) document. Table 43-2 provides details.

43.1.1.1.3 Opaque Passed by File/FTP Adapters

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 43-3 provides details.

43.1.1.1.4 Opaque Passed by Oracle B2B

In this use case, the binary attachments (for example, an image) are Base64-encoded as text encoded. Table 43-4 provides details.

43.1.1.2.1 SOAP with Attachments

In this use case, the binary attachments (for instance an image) are passed end-to-end as a stream. Table 43-5 provides details.

<mime:part>
   <mime:content part="bin" type=“image/jpeg"/>
</mime:part>

Working with Streaming Attachments

Oracle Fusion Middleware web services enable you to pass large attachments as a stream. Unlike the JAX-RPC API, which treats attachments as if they are entirely in memory, streams make the programming model more efficient to use. Streams also enhance performance and scalability because there is no need to load the attachment into memory before service execution.

As with embedded attachments, streamed attachments conform to the multipart MIME binary format. On the wire, messages with streamed attachments are identical to any other SOAP message with attachments.

Example 43-1 provides a sample message with a streamed attachment. The first part in the message is the SOAP envelope (<SOAP-ENV:Envelope...). The second part is the attachment (for this example, myImage.gif).

MIME-Version: 1.0
Content-Type: Multipart/Related; boundary=MIME_boundary; type=text/xml;
Content-Description: This is the optional message description.

--MIME_boundary
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-ID: NotSure/DoesntMatter

<?xml version='1.0' ?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Body>
. . .
<DocumentName>MyImage.gif</DocumentName>
. . .
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

--MIME_boundary
Content-Type: image/gif
Content-Transfer-Encoding: binary
Content-ID: AnythingYoudLike

...binary GIF image...
--MIME_boundary--

Creating Composites that Use MIME Attachments

Perform the following procedures to create composites that use MIME attachments.

To create composites that use MIME attachments:

1. Create a composite using a payload schema (for example, an inbound web service wired to an Oracle Mediator wired to an outbound web service).

2. Within the WSDL file of Oracle Mediator, perform the following steps:

   1. From the WSDL designer, open the Oracle Mediator WSDL file.

   2. Drag and drop bindings into the middle swimlane.

   3. Select the RPC binding.

   4. Enter a name.

   5. Go to Source view of the WSDL and modify the WSDL input and WSDL output with MIME multiparts.

      <wsdl:input>
            <mime:multipartRelated>
                <mime:part>
                    <soap:body parts="payload" use="literal"/>
                </mime:part>
                <mime:part>
                    <mime:content part="bin" type="application/octet-stream"/>
                </mime:part>
            </mime:multipartRelated>
      </wsdl:input>
      

   6. Add the MIME part in the request/response message.

      <wsdl:message name="BPELProcess1RequestMessage">
          <wsdl:part name="payload" element="ns1:purchaseOrder" />
          <!--add below part-->
          <wsdl:part name="bin" type="xsd:base64Binary"/>
      </wsdl:message>
      

   7. Add a namespace in the WSDL definitions.

      xmlns:xsd="http://www.w3.org/2001/XMLSchema"
      xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/">
      

When complete, the WSDL that references a MIME attachment is displayed.

<wsdl:definitions
  name="PhotoCatalogService"
  targetNamespace="http://examples.com/PhotoCatalog"
  xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:types="http://examples.com/PhotoCatalog/types"
  xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/"
  xmlns:tns="http://examples.com/PhotoCatalog">
  <wsdl:message name="addPhotoRequest">
    <wsdl:part name="photo" type="xsd:hexBinary"/>
  </wsdl:message>
  <wsdl:message name="addPhotoResponse">
    <wsdl:part name="status" type="xsd:string"/>
  </wsdl:message>
  <wsdl:message name="replacePhotoRequest">
    <wsdl:part name="oldPhoto" type="xsd:string"/>
    <wsdl:part name="newPhoto" type="xsd:hexBinary"/>
  </wsdl:message>
  <wsdl:message name="replacePhotoResponse">
    <wsdl:part name="status" type="xsd:string"/>
  </wsdl:message>
  <wsdl:portType name="PhotoCatalog">
    <wsdl:operation name="addPhoto">
      <wsdl:input message="tns:addPhotoRequest"/>
      <wsdl:output message="tns:addPhotoResponse"/>
    </wsdl:operation>
    <wsdl:operation name="replacePhoto">
      <wsdl:input message="tns:replacePhotoRequest"/>
      <wsdl:output message="tns:replacePhotoResponse"/>
    </wsdl:operation>
  </wsdl:portType>
  <wsdl:binding name="PhotoCatalogBinding" type="tns:PhotoCatalog">
    <soap:binding style="document"
transport="http://schemas.xmlsoap.org/soap/http"/>
    <wsdl:operation name="addPhoto">
      <wsdl:input>
        <mime:multipartRelated>
          <mime:part>
            <soap:body use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="photo"
                          type="image/jpeg"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:input>
      <wsdl:output>
        <mime:multipartRelated>
          <mime:part>
            <soap:body use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="status" type="text/plain"/>
            <mime:content part="status" type="text/xml"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:output>
    </wsdl:operation>
    <wsdl:operation name="replacePhoto">
      <wsdl:input>
        <mime:multipartRelated>
          <mime:part>
            <soap:body parts="oldPhoto" use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="newPhoto"
                          type="image/jpeg"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:input>
      <wsdl:output>
            <soap:body parts="status" use="literal"/>
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>
</wsdl:definitions>

Performance Overhead and Pass Through Attachments

Because Oracle Mediator is stateless, there is no performance overhead with pass through attachments. However, Oracle BPEL Process Manager dehydrates attachments and has performance overhead, even for pass through attachments. Using Oracle BPEL Process Manager for attachments over a period of time, the SOA Infrastructure schema can grow to its maximum size and encounter memory issues. It is recommended that you extend the database tablespace appropriately for the SOA Infrastructure schema to accommodate large attachments. Simultaneously, you can use purge scripts to purge completed instances along with the attachments table. For information on purge scripts, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle BPM Suite.

In scenarios in which one BPEL process calls a second BPEL process within the same composite, the second BPEL process does not dehydrate the same attachment again.

In scenarios in which one BPEL process from composite 1 invokes a second BPEL process from composite 2 and optimization is disabled, composite 1 makes a SOAP call to composite 2. The second BPEL process does dehydrate attachments.

Properties for Streaming Attachments

To stream attachments, add the following properties in the composite.xml file. If optimization is enabled, then a native call is used instead of a SOAP call. Example 43-2 provides details.

<binding.ws
port="http://services.otn.com#wsdl.endpoint(MIMEService/MIMEService)"
xmlns:ns="http://xmlns.oracle.com/sca/1.0"
streamIncomingAttachments="true" streamOutgoingAttachments="true">
<!--Add this prop to reference bindings to make a SOAP call. -->
<property name="oracle.webservices.local.optimization">false</property>
</binding.ws>

For information about the oracle.webservices.local.optimization property, see "Managing SOA Composite Application Policies" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle BPM Suite.

Reading and Encoding SOAP Attachment Content

The ora:getAttachmentContent function reads SOAP attachment content and encodes that data in Base64 format in a BPEL process by providing the BPEL variable as an argument, which has an href of the SOAP attachment. Example 43-3 shows how to use this function:

<copy>
   <from expression="ora:getAttachmentContent('input','bin')"/>
   <to variable="initiateTaskInput" part="payload"
    query="/taskservice:initiateTask/task:task/task:attachment/task:content"/>
</copy>

Example 43-3 copies the attachment content, which has its href stored in the "input/bin" variable to the content variable, in Base64-encoded format.

Sending Attachment Streams

Oracle Mediator can pass an attachment stream to only one target receiver. The receiver can be another component or a web service/adapter. The second target cannot receive the attachment. Oracle BPEL Process Manager supports sending the attachment stream to multiple receivers. For Oracle BPEL Process Manager to send a stream to multiple receivers, it must read the attachment stream from the database using the readBinaryFromFile XPath function and pass the stream to the appropriate targets.

Sharing Attachments Using Synchronous Flows

When Oracle BPEL Process Manager-based composites share attachments using synchronous flows, it is necessary to use the same end-to-end transaction. This is applicable to composites that are colocated and use local/optimized calls. This can be achieved by setting the property shown in Example 43-4 on all the called BPEL components (callees) in the call chain.

<property name="bpel.config.transaction" many="false"
type="xs:string">required</property>

If such composites do not execute as part of the same transaction context, the attachment data saved by the first BPEL component in the call chain is not visible to the other BPEL components in the call chain. In addition, they incur database locking and timeout exceptions:

"ORA-02049: timeout: distributed transaction waiting for lock"

43.1.1.2.2 Attachment Options of File/FTP Adapters

In this use case, the adapter streams the binary data to a database store and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator). Table 43-6 provides details.

Writing Attachments Using an Outbound File Adapter

Example 43-5 shows a sample schema that can be used by the file adapter to write attachments to disk.

<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns="http://xmlns.oracle.com/attachment"
            targetNamespace="http://xmlns.oracle.com/attachment"
            elementFormDefault="qualified">
            <xsd:element name="attach">
              <xsd:complexType>
                 <xsd:attribute name="href" type="xsd:string"/>
              </xsd:complexType>
  </xsd:element>
</xsd:schema>

Use Oracle Mediator in the flow to map the attachment part from the source (Oracle Mediator) to the target (file adapter) using an Oracle Mediator assign.

If you use Oracle BPEL Process Manager, the attachment is written to the dehydration store, which slows down the process.

Transforming Attachments with the ora:doStreamingTranslate XPath Function

Use of the ora:doStreamingTranslate XPath function is only recommended while transforming attachments within an Oracle BPEL Process Manager/Oracle Mediator composite. This function expects the attachment location to be a relative path on the server. This function cannot translate incoming attachment streams.

For more information about this function, see Section B.2.6, "doStreamingTranslate."

43.1.1.2.3 Oracle B2B Attachment

In this use case, Oracle B2B stores the binary data to a database and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator) based on an Oracle B2B-defined XSD. Oracle B2B protocols define the attachment. Table 43-7 provides details.

43.1.1.3.1 Outbound Composite SOAP Messages Are Not Optimized If Only a WS-MTOM Policy is Used

Unless a SOAP message passed to the dispatch.invoke() call of the SOA Infrastructure already contains BinaryTextIml nodes, binary data transmission is not optimized on the wire. Therefore, there is no guarantee that binary data optimization is always performed when a WS-MTOM policy is configured. The only way to ensure that optimization is performed is if the SOA MTOM configuration is also specified. The WS-MTOM policy guarantees the proper content-type setting with or without the SOA MTOM settings.

For example, assume you create a SOA composite application without BinaryTextIml nodes in the SOAP message that consists of the following components:

• A Java API for an XML Web Services (JAX-WS), MTOM-enabled, client service binding component

• An MTOM-enabled BPEL service component

• A JAX-WS web service reference binding component with MTOM

The JAX-WS, MTOM-enabled, client service binding component invokes the BPEL service component. The BPEL service component then invokes the JAX-WS, web service, reference binding component. The SOAP message from the JAX-WS client service binding component to the BPEL service component is MTOM-optimized. However, from the BPEL service component to the JAX-WS, web service reference binding component, the message is base64binary-enabled, and not MTOM-optimized.

43.1.1.4.1 Debatching with the File/FTP Adapter

In this use case, the inbound adapter splits a source document into multiple batches of records, each of which initiates a composite instance. Table 43-8 provides details.

43.1.1.4.2 Chunking with the File/FTP Adapters

In this use case, a loop within a BPEL process reads a chunk of records at a time and process (that is, cursor). Table 43-9 provides details.

43.1.1.5.1 Streaming with the File/FTP Adapters

In this use case, very large XML files are streamed through Oracle SOA Suite. Table 43-10 provides details.

43.1.1.5.2 Oracle B2B Streaming

In this use case, large XML files are passed by Oracle B2B to Oracle SOA Suite as an SDOM. This only occurs when a large payload size is defined in the Oracle B2B user interface. Table 43-11 provides details.

43.1.3.6.1 MDSInstance Cache Size

To set Metadata Service (MDS) instance cache size, the property and value must be added to the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file, as shown in Example 43-6.

<property>
 <name>b2b.mdsCache</name>
 <value>200000</value>
 <comment>MDS Instance cache size </comment>
</property>

43.1.3.6.2 Protocol Message Size

If Oracle B2B wants to send or receive more than 10 MB of message or the import/export configuration is more than 10 MB, then change the following setting accordingly at the Oracle WebLogic Server Administration Console:

1. In the Domain Structure, select Environment > Servers.

2. In the Name column of the table, select soa_server.

3. Select the Protocols tab.

4. Change the value for Maximum Message Size.

This setting can also be added/modified in the $DOMAIN_HOME/config/config.xml file next to the server name configuration, as shown in Example 43-7.

<name>soa_server1</name>
<max-message-size>150000000</max-message-size> 

43.1.3.6.3 Number of Threads

This parameter helps to improve the message processing capability of Oracle B2B and must be set in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file. Example 43-8 provides an example.

<property>
  <name>b2b.inboundProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.inboundProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property>
<property>
  <name>b2b.outboundProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.outboundProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property>
<property>
  <name>b2b.defaultProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.defaultProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property> 

43.1.3.6.4 Stuck Thread Max Time

The Stuck Thread Max Time parameter checks the number of seconds that a thread must be continually working before the server considers the thread stuck. You must change the following setting in the Oracle WebLogic Server Administration Console:

1. In the Domain Structure, select Environment > Servers.

2. In the Name column of the table, select soa_server.

3. Select the Tuning tab.

4. Change the value for Stuck Thread Max Time.

43.1.3.6.5 Tablespace

If you must store more than a 150 MB configuration in the data file, then you must extend or add the data file to increase the tablespace size, as shown in Example 43-9.

ALTER TABLESPACE sh_mds add DATAFILE 'sh_mds01.DBF' SIZE 100M autoextend on next
 10M maxsize unlimited;
ALTER TABLESPACE sh_ias_temp add TEMPFILE 'sh_ias_temp01.DBF' SIZE 100M autoextend
 on next 10M maxsize unlimited;