JMS XML

25 JMS XML

It is important to understand how to work with Java Message Services (JMS) with a XML payload in Oracle Data Integrator.

This chapter includes the following sections:

25.1 Introduction

Oracle Data Integrator provides a simple and transparent method to integrate JMS destinations. This chapter focuses on processing JMS messages with a XML payload. For text payload processing in batch mode, refer to JMS.

25.1.1 Concepts

The JMS XML Knowledge Modules apply to most popular JMS compliant middleware, including Oracle Service Bus, Sonic MQ, and so forth. Most of these Knowledge Modules include transaction handling to ensure message delivery.

25.1.1.1 JMS Message Structure

See JMS Message Structure for information about the JMS message structure.

25.1.1.2 Using a JMS Destination

Oracle Data Integrator is able to process XML messages that are delivered by a JMS destination. Each message is considered as a container for XML data and is handled through the JMS XML Queue or JMS XML Topic technology.

With JMS XML Queue/JMS XML Topic technologies, each messages payload contains a complete XML data structure. This structure is mapped into a relational schema (XML Schema) that appears as a model, using the Oracle Data Integrator XML Driver.

Note:

This method is extremely similar to XML files processing. In JMS XML, the message payload is the XML file. See XML Filesand Oracle Data Integrator Driver for XML Referencefor more information about XML Files processing and the XML Driver.

In the topology, each JMS destination is defined as a JMS XML Topic/Queue data server with a single physical schema. A data server/physical schema pair will be declared for each topic or queue delivering message in the XML format.

The structure of the XML message mapped into a relational structure (called the XML schema) appears as a data model. Each datastore in this model represents a portion (typically, an element type) in the XML file.

Processing Messages

As each XML message corresponds to an Oracle Data Integrator model, the entire model must be used and loaded as one single unit when a JMS XML message is consumed or produced. The processing unit for an XML message is the package.

It is not possible to declare the properties or header fields of the message in the model or use them as attributes in a mapping. They still can be used in message selectors, or be set through KM options.

Consuming an XML message

Processing an incoming XML message is performed in packages as follows:

Synchronize the JMS message to the XML schema: This operation reads the message and generates the XML schema. This is usually performed by the first mapping accessing the message.
Extract the data: A sequence of mappings use datastores from the XML schema as sources. This data is usable until the session is terminated, another message is read by a new Synchronize action, or the Commit JMS Read is performed.
Commit JMS Read: This operation validates the message consumption and deletes the XML schema. It should be performed by the last mapping which extracts data from the XML message.

Producing an XML message

To produce an XML message, a package must be designed to perform the following tasks:

Initialize the XML schema: This operation creates an empty XML schema corresponding to the XML message to generate. This operation is usually performed in the first mapping loading the structure.
Load the data: A sequence of mappings loads data into the XML schema.
Synchronize the XML schema to JMS: This operation converts the XML schema to an XML message, and sends it to the JMS destination. This operation is usually performed by the last mapping loading the schema.

Filtering Messages

It is possible to filter messages from a JMS XML destination by defining a Message Selector (MESSAGE_SELECTOR KM option) to filter messages on the server. This type of filter can use only the properties or header fields of the message. The filter is processed by the server, reducing the amount of information read by Data Integrator. It is also possible to filter data in the mapping using data extracted from the XML schema. These filters are processed in Data Integrator, after the message is synchronized to the XML schema.

25.1.2 Knowledge Modules

Oracle Data Integrator provides the Knowledge Modules (KM) listed in Table 25-1 for handling XML messages.

Table 25-1 JMS XML Knowledge Modules

Knowledge Module	Description
IKM SQL to JMS XML Append	Integrates data into a JMS compliant message queue or topic in XML format from any ANSI SQL-92 standard compliant staging area.
LKM JMS XML to SQL	Loads data from a JMS compliant message queue or topic in XML to any ANSI SQL-92 standard compliant database used as a staging area.

25.2 Installation and Configuration

Make sure you have read the information in this section before you start using the JMS Knowledge Modules:

25.2.1 System Requirements and Certifications

Before performing any installation you should read the system requirements and certification documentation to ensure that your environment meets the minimum installation requirements for the products you are installing.

The list of supported platforms and versions is available on Oracle Technical Network (OTN):

http://www.oracle.com/technetwork/middleware/data-integrator/documentation/index.html.

25.2.2 Technology Specific Requirements

The JMS destinations are usually accessed via a JNDI service. The configuration and specific requirements for JNDI and JMS depends on the JMS Provider you are connecting to. Refer to the JMS Provider specific documentation for more details.

Note:

By default, a sequence of four ';' is used as fixed record separator for JMS XML driver read operations. If the XML data also contains a sequence of four or more ';' characters, an error will occur and you must set the record separator to a different value. This is achieved using the Doracle.odi.jmsxmlColSepString JVM option. For example, Doracle.odi.jmsxmlColSepString="????" will set the JMS XML driver record separator to "????" instead of ";;;;".

This option must be set in the following locations:

In Studio, this parameter is set in the odi.conf parameter file. Add a new AddVMOption entry.
For 12c Standalone/Colocated Agents, use ODI_INSTANCE_OPTIONS in the instance.sh script.
For 11g Standalone Agents, use ODI_ADDITIONAL_JAVA_OPTIONS in the odiparams file.
For JEE Agents, add it to JAVA_OPTIONS in the startManagedWeblogic script.

25.2.3 Connectivity Requirements

This section lists the requirements for connecting to a JMS XML database.

Oracle Data Integrator does not include specific drivers for JMS providers. Refer to the JMS Provider documentation for the connectivity requirement of this provider.

XML Configuration

XML content is accessed through the Oracle Data Integrator JDBC for XML driver. The driver is installed with Oracle Data Integrator.

Ask your system administrator for the location of the DTD file describing the XML content.

25.3 Setting up the Topology

Setting up the Topology consists of:

25.3.1 Creating a JMS XML Data Server

An JMS XML data server corresponds to one JMS provider/router that is accessible through your local network.

There are two types of JMS XML data servers: JMS Queue XML and JMS Topic XML.

A JMS Queue XML data server is used to connect a single queue in the JMS router to integrate XML messages.
A JMS Topic XML data server is used to connect a single Topic in the JMS router to integrate XML messages.

The Oracle Data Integrator JMS driver loads the messages that contains the XML content into a relational schema in memory. This schema represents the hierarchical structure of the XML message and enables unloading the relational structure back to the JMS messages.

25.3.1.1 Creation of the Data Server

Create a data server either for the JMS Queue XML technology or for the JMS Topic XML technology using the standard procedure, as described in Creating a Data Server of Administering Oracle Data Integrator.

The creation process for a JMS XML Queue or JMS Topic XML data server is identical to the creation process of an XML data server except that you need to define a JNDI connection with JMS XML specific information in the JNDI URL. See Creating an XML Data Server for more information.

This section details only the fields required or specific for defining a JMS Queue XML or JMS Topic XML data server.

In the Definition tab:
- Name: Name of the data server as it will appear in Oracle Data Integrator.
- User/Password: Not used here. Leave these fields empty.

In the JNDI tab:

JNDI Authentication: From the list, select the authentication mode.
JNDI User: Enter the username to connect to the JNDI directory (not mandatory).
Password: This user's password (not mandatory).
JNDI Protocol: From the list, select the JNDI protocol (not mandatory).
JNDI Driver: Name of the initial context factory java class to connect to the JNDI provider, for example:

com.sun.jndi.ldap.LdapCtxFactory
JNDI URL: <JMS_RESOURCE>?d=<DTD_FILE>&s=<SCHEMA>&JMS_DESTINATION=<JMS_DESTINATION_NAME>.

The JNDI URL properties are described inTable 25-2.

JNDI Resource: Logical name of the JNDI resource corresponding to your JMS Queue (or Topic) connection factory.

Note:

Specify QueueConnectionFactory if you want to access a message queue and TopicConnectionFactory if you want to access a topic. Note that these parameters are specific to the JNDI directory.

Table 25-2 JNDI URL Properties

Parameter	Value	Notes
d	<DTD File location>	DTD File location (relative or absolute) in UNC format. Use slash "/" in the path name and not backslash "\" in the file path. This parameter is mandatory.
re	<Root element>	Name of the element to take as the root table of the schema. This value is case sensitive. This parameter can be used for reverse-engineering a specific message definition from a WSDL file, or when several possible root elements exist in a XSD file.
ro	true \| false	If true, the XML file is opened in read only mode.
s	<schema name>	Name of the relational schema where the XML file will be loaded.This value must match the one set for the physical schema attached to this data server. This parameter is mandatory.
cs	true \| false	Load the XML file in case sensitive or insensitive mode. For case insensitive mode, all element names in the DTD file should be distinct (Ex: Abc and abc in the same file are banned). The case sensitive parameter is a permanent parameter for the schema. It CANNOT be changed after schema creation. Please note that when opening the XML file in insensitive mode, case will be preserved for the XML file.
JMSXML_ROWSEPARATOR	5B23245D	Hexadecimal code of the string used as a line separator (line break) for different XML contents. Default value is 5B23245D which corresponds to the string [#$].
JMS_DESTINATION	JNDI Queue name or Topic name	JNDI Name of the JMS Queue or Topic. This parameter is mandatory.
transform_nonascii or tna	boolean (true\|false)	Transform Non Ascii. Set to false to keep non-ascii characters. Default is true. This parameter is not mandatory.

Example 25-1 Example

If using an LDAP directory as the JNDI provider, you should use the following parameters:

JNDI Driver: com.sun.jndi.ldap.LdapCtxFactory
JNDI URL: ldap://<ldap_host>:<port>/<dn>?d=<DTD_FILE>&s=<SCHEMA>&JMS_DESTINATION=<JMS_DESTINATION_NAME>
JNDI Resource: <Name of the connection factory>

25.3.2 Creating a JMS XML Physical Schema

Create a JMS XML physical schema using the standard procedure, as described in Creating a Physical Schema in Administering Oracle Data Integrator.

Note:

For the name of the Schema and Work Schema use the schema name defined in the s=<schema name> property of the JNDI URL of the JMS Queue XML or JMS Topic XML data server.

Note:

Only one physical schema is required per JMS XML data server.

Create for this physical schema a logical schema using the standard procedure, as described in Creating a Logical Schema in Administering Oracle Data Integrator and associate it in a given context.

25.4 Setting Up an Integration Project

Setting up a project using JMS XML follows the standard procedure. See Creating an Integration Project of Developing Integration Projects with Oracle Data Integrator.

It is recommended to import the following knowledge modules into your project for getting started with JMS XML:

IKM SQL to JMS XML Append
LKM JMS XML to SQL

25.5 Creating and Reverse-Engineering a JMS XML Model

This section contains the following topics:

25.5.1 Create a JMS XML Model

Create a JMS Queue XML or JMS Topic XML Model using the standard procedure, as described in Creating a Model of Developing Integration Projects with Oracle Data Integrator.

A JMS Queue XML or JMS Topic XML Model corresponds to a set of datastores, with each datastore representing an entry level in the XML file. The models contain datastores describing the structure of the JMS messages. A model contains the message structure of one topic or one queue. This model's structure is reverse-engineered from the DTD or the XML file specified in the data server definition, using standard reverse-engineering.

25.5.2 Reverse-Engineering a JMS XML Model

JMS XML supports Standard reverse-engineering - which uses only the abilities of the XML driver.

To perform a Standard Reverse-Engineering on JMS Queue XML or JMS Topic XML use the usual procedure, as described in Reverse-engineering a Model of Developing Integration Projects with Oracle Data Integrator.

Oracle Data Integrator will automatically add the following attributes to the datastores generated from the XML data:

Primary keys (PK attributes) for parent-child relationships
Foreign keys (FK attributes) for parent-child relationships
Order identifier (ORDER attributes) to enable the retrieval of the order in which the data appear in the XML file.

These extra attributes enable the hierarchical XML structure's mapping in a relational structure stored in the schema. See Oracle Data Integrator Driver for XML Reference for more information.

25.6 Designing a Mapping

The KM choice for a mapping or a check determines the abilities and performance of this mapping or check. The recommendations in this section help in the selection of the KM for different situations concerning XML messages.

25.6.1 Loading Data from a JMS XML Source

JMS XML can be used as a source or a target in a mapping. Data from an XML message Queue or Topic can be loaded to any ANSI SQL-92 standard compliant database used as a staging area. The LKM choice in the Mapping Flow tab to load data between JMS XML and another type of data server is essential for successful data extraction.

Oracle Data Integrator provides the LKM JMS XML to SQL for loading data from a JMS compliant message queue or topic in XML to any ANSI SQL-92 standard compliant database used as a staging area. This LKM uses the Agent to read selected messages from the source queue/topic and write the result in the staging temporary table created dynamically.To ensure message delivery, the message consumer (or subscriber) does not commit the read until the data is actually integrated into the target by the IKM.Consider using this LKM if one of your source datastores is an XML JMS message.

In order to load XML messages from a JMS provider, the following steps must be followed:

The first mapping reading the XML message from the JMS XML source must use the LKM JMS XML to SQL with the SYNCHRO_JMS_TO_XML LKM option set to Yes. This option creates and loads the XML schema from the message retrieved from the queue or topic.
The last mapping should commit the message consumption by setting the JMS_COMMIT to Yes.

Table 25-3 lists the JMS specific options of this knowledge module.

25.6.2 Integrating Data in a JMS XML Target

Oracle Data Integrator provides the IKM SQL to JMS XML Append that implements optimized data integration strategies for JMS XML. This IKM integrates data into a JMS compliant message queue or topic in XML format from any ANSI SQL-92 standard compliant staging area.

To use this IKM, the staging area must be different from the target.

In order to integrate XML data into a JMS XML target, the following steps must be followed:

The first mapping loading the XML schema must provide a value for the ROOT_TABLE (it is the model's table that corresponds to the root element of the XML file), and also set the INITIALIZE_XML_SCHEMA option to Yes.

Note:

The root table of the XML schema usually corresponds to the datastore at the top of the hierarchy tree view of the JMS XML model. Therefore the ROOT_TABLE parameter should take the value of the resource name for this datastore.
The mappings should load the datastores in the hierarchy order, starting by the top of the hierarchy and going down. The mappings loading subsequent levels of the XML schema hierarchy should load the foreign key attribute linking the current hierarchy level to a higher one.

For example, when loading the second level of the hierarchy (the one under the root table), the foreign key attribute should be set to '0' (Zero), as it is the value that is set by the IKM in the root table primary key when the root table is initialized.
The last mapping should send the XML schema to the JMS provider by setting the SYNCHRO_JMS_TO_XML parameter to Yes.

Example

An XML file format generates a schema with the following hierarchy of datastores:

+ GEOGRAPHY_DIM (GEO_DIMPK, ...)
  |
  +--- COUNTRY (GEO_DIMFK, COUNTRYPK, COUNTRY_NAME, ...)
       |
       +--- REGION (COUNTRYFK, REGIONPK, REGION_NAME, ...)

In this hierarchy, GEOGRAPHY_DIM is the root table, and its GEOGRAPHY_DIMPK attribute is set to '0' at initialization time. The tables should be loaded in the GEOGRAPHY_DIM, COUNTRY, REGION sequence.

When loading the second level of the XML hierarchy (COUNTRY) make sure that the FK field linking this level to the root table level is set to '0'. In the model above, when loading COUNTRY, we must load the COUNTRY.GEOGRAPHY_DIMFK set to '0'.
You must also link the records of REGION to the COUNTRY level by loading the REGION.COUNTRYFK attribute with values that correspond to a parent record in COUNTRY (having REGION.COUNTRYFK = COUNTRY.COUNTRYPK).

For more information on loading data to XML schemas, see Oracle Data Integrator Driver for XML Reference.

Table 25-3 lists the JMS specific KM options of this IKM. Options that are specific to XML messages are in bold.

JMS XML Knowledge Modules Options

Table 25-3 lists the KM options for the LKM and IKM for JMS XML. Options that are specific to XML messages are in bold.

Although most options are the same for the LKM and IKM, there are only few differences:

The INITIALIZE_XML_SCHEMA and ROOT_TABLE options are only provided for the IKM.
The DELETE_TEMPORARY_OBJECTS and JMS_COMMIT options are only provided for the LKM.
Set JMS_COMMIT to Yes to commit the message consumption on the Router (JMS XML).

Table 25-3 JMS Specific KM Options

Option	Used to	Description
CLIENTID	Read	Subscriber identification string. Not used for publication.
DURABLE	Read	D: Session is durable. Indicates that the subscriber definition remains on the router after disconnection.
INITIALIZE_XML_SCHEMA	Write	Initializes an empty XML schema. This option must be set to `YES` for the first mapping loading the schema.
JMSDELIVERYMODE	Write	JMS delivery mode (1: Non Persistent, 2: Persistent). A persistent message remains on the server and is recovered on server crash.
JMSEXPIRATION	Write	Expiration delay in milliseconds for the message on the server [0..4 000 000 000]. 0 signifies that the message never expires. Warning! After this delay, a message is considered as expired, and is no longer available in the topic or queue. When developing mappings it is advised to set this parameter to zero.
JMSPRIORITY	Write	Relative Priority of the message: 0 (lowest) to 9 (highest).
JMSTYPE	Write	Optional name of the message.
MESSAGEMAXNUMBER	Read	Maximum number of messages retrieved [0 .. 4 000 000 000]. 0: All messages are retrieved.
MESSAGESELECTOR	Read	Message selector in ISO SQL syntax for filtering on the router. See Using JMS Properties for more information on message selectors.
MESSAGETIMEOUT	Read	Time to wait for the first message in milliseconds [0 .. 4 000 000 000]. If MESSAGETIMEOUT is equal to 0, then there is no timeout. MESSAGETIMEOUT and MESSAGEMAXNUMBER cannot be both equal to zero. If `MESSAGETIMEOUT= 0` and `MESSAGEMAXNUMBER =0`, then MESSAGETIMEOUT takes the value 1. Warning! A mapping may retrieve no message if this timeout value is too small.
NEXTMESSAGETIMEOUT	Read	Time to wait for each subsequent message in milliseconds [0 .. 4 000 000 000]. The default value is 1000. Warning! A mapping may retrieve only part of the messages available in the topic or the queue if this value is too small.
ROOT_TABLE	Write	Resource name of the datastore that is the root of the XML model hierarchy. Option applicable only to first mapping loading the schema (INITIALIZE_XML_SCHEMA=true). IKM inserts a record for the root element of the XML schema, if ROOT_TABLE<>'' and INITIALIZE_XML_SCHEMA=true. Warning! Use only, if no mapping will populate the root table of the XML structure. Otherwise a duplicate root element will be encountered.
SENDMESSAGETYPE	Write	Type of message to send (1 -> BytesMessage, 2 ->TextMessage).
SYNCHRO_XML_TO_JMS	Write	Generates the XML message from the XML schema, and sends this message. This option must be set to `YES` for the last mapping that writes to the schema XML.