21 Monitoring Integration Processes

This chapter describes how to manage your development executions in Operator Navigator. An overview of the Operator Navigator's user interface is provided.

This chapter includes the following sections:

Section 21.1, "Introduction to Monitoring"
Section 21.2, "Monitoring Executions Results"
Section 21.3, "Managing your Executions"

21.1 Introduction to Monitoring

Monitoring your development executions consists of viewing the execution results and managing the development executions when the executions are successful or in error. This section provides an introduction to the monitoring features in Oracle Data Integrator. How to work with your execution results is covered in Section 21.2, "Monitoring Executions Results". How to manage your development executions is covered in Section 21.3, "Managing your Executions".

21.1.1 Introduction to Operator Navigator

Through Operator Navigator, you can view your execution results and manage your development executions in the sessions, as well as the scenarios and Load Plans in production.

Operator Navigator stores this information in a work repository, while using the topology defined in the master repository.

Operator Navigator displays the objects available to the current user in six accordions:

Session List displays all sessions organized per date, physical agent, status, keywords, and so forth
Hierarchical Sessions displays the execution sessions also organized in a hierarchy with their child sessions
Load Plan Executions displays the Load Plan Runs of the Load Plan instances
Scheduling displays the list of physical agents and schedules
Load Plans and Scenarios displays the list of scenarios and Load Plans available
Solutions displays the list of solutions

The Operator Navigator Toolbar Menu

You can perform the main monitoring tasks via the Operator Navigator Toolbar menu. The Operator Navigator toolbar menu provides access to the features detailed in Table 21-1.

Table 21-1 Operator Navigator Toolbar Menu Items

Icon	Menu Item	Description
	Refresh	Click Refresh to refresh the trees in the Operator Navigator accordions.
	Filter Filter activated	Click Filter to define the filters for the sessions to display in Operator Navigator.
	Auto Refresh	Click Auto Refresh to refresh automatically the trees in the Operator Navigator accordions.
	Connect Navigator	Click Connect Navigator to access the Operator Navigator toolbar menu. Through the Operator Navigator toolbar menu you can: Import a scenario Import and export the log Perform multiple exports Purge the log Display the scheduling information Clean stale sessions

21.1.2 Scenarios

A scenario is designed to put a source component (interface, package, procedure, variable) into production. A scenario results from the generation of code (SQL, shell, etc.) for this component.

When a scenario is executed, it creates a Session.

Scenarios are imported into production environment and can be organized into Load Plan and Scenario folders. See Section 21.3.4, "Managing Scenarios and Load Plans" for more details.

21.1.3 Sessions

In Oracle Data Integrator, an execution results in a Session. Sessions are viewed and managed in Operator Navigator.

A session is an execution (of a scenario, an interface, a package or a procedure, and so forth) undertaken by an execution agent. A session is made up of steps which are themselves made up of tasks.

A step is the unit of execution found between a task and a session. It corresponds to a step in a package or in a scenario. When executing an interface or a single variable, for example, the resulting session has only one step.

Two special steps called Command On Connect and Command On Disconnect are created if you have set up On Connect and Disconnect commands on data servers used in the session. See Setting Up On Connect/Disconnect Commands for more information.

The task is the smallest execution unit. It corresponds to a command in a KM, a procedure, and so forth.

Sessions can be grouped into Session folders. Session folders automatically group sessions that were launched with certain keywords. Refer to Section 21.3.3.3, "Organizing the Log with Session Folders" for more information.

21.1.4 Load Plans

A Load Plan is the largest executable object in Oracle Data Integrator. It uses Scenarios in its steps. A Load Plans is an organized hierarchy of child steps. This hierarchy allows conditional processing of steps in parallel or in series.

Load Plans are imported into production environment and can be organized into Load Plan and Scenario folders. See Section 21.3.4, "Managing Scenarios and Load Plans" for more details.

21.1.5 Load Plan Executions

Executing a Load Plan creates a Load Plan instance and the first Load Plan run for the instance. This Load Plan instance is separated from the original Load Plan and can be modified independently. Every time a Load Plan instance is restarted, a Load Plan run is created for this Load Plan instance. A Load Plan run corresponds to an attempt to execute the instance. See Section 14.1.1, "Load Plan Execution Lifecycle" for more information.

When running, a Load Plan Run starts sessions corresponding to the scenarios sequenced in the Load Plan.

Note that in the list of Load Plan executions, only the Load Plan runs appear. Each run is identified by a Load Plan Instance ID and an Attempt (or Run) Number.

21.1.6 Schedules

You can schedule the executions of your scenarios and Load Plans using Oracle Data Integrator's built-in scheduler or an external scheduler. Both methods are detailed in Section 20.9, "Scheduling Scenarios and Load Plans".

The schedules appear in Designer and Operator Navigator under the Scheduling node of the scenario or Load Plan. Each schedule allows a start date and a repetition cycle to be specified.

21.1.7 Log

The Oracle Data Integrator log corresponds to all the Sessions and Load Plan instances/runs stored in a repository. This log can be exported, purged or filtered for monitoring. See Section 21.3.3, "Managing the Log" for more information.

21.1.8 Status

A session, step, task or Load Plan run always has a status. Table 21-2 lists the six possible status values:

Table 21-2 Status Values

Status Name	Status Icon for Sessions	Status Icon for Load Plans	Status Description
Done			The Load Plan, session, step or task was executed successfully.
Done in previous run			The Load Plan step has been executed in a previous Load Plan run. This icon is displayed after a restart.
Error			The Load Plan, session, step or task has terminated due to an error.
Running			The Load Plan, session, step or task is being executed.
Waiting			The Load Plan, session, step or task is waiting to be executed.
Warning (Sessions and tasks only)			For Sessions: The session has completed successfully but errors have been detected during the data quality check. For Tasks: The task has terminated in error, but since errors are allowed on this task, this did not stop the session.
Queued (Sessions only)			The session is waiting for an agent to be available for its execution

When finished, a session takes the status of the last executed step (Done or Error). When finished, the step, takes the status of the last executed task (Except if the task returned a Warning. In this case, the step takes the status Done).

A Load Plan is successful (status Done) when all its child steps have been executed successfully. It is in Error status when at least one of its child steps is in error and has raised its exception to the root step.

21.2 Monitoring Executions Results

In Oracle Data Integrator, an execution results in a session or in a Load Plan run if a Load Plan is executed. A session is made up of steps which are made up of tasks. Sessions are viewed and managed in Operator Navigator.

Load Plan runs appear in the Operator Navigator. To review the steps of a Load Plan run, you open the editor for this run. The sessions attached to a Load Plan appear with the rest of the sessions in the Operator Navigator.

21.2.1 Monitoring Sessions

To monitor your sessions:

In the Operator Navigator, expand the Session List accordion.
Expand the All Executions node and click Refresh in the Navigator toolbar.
Optionally, activate a Filter to reduce the number of visible sessions. For more information, see Section 21.3.3.1, "Filtering Sessions".
Review in the list of sessions the status of your session(s).

21.2.2 Monitoring Load Plan Runs

To monitor your Load Plan runs:

In the Operator Navigator, expand the Load Plan Executions accordion.
Expand the All Executions node and click Refresh in the Navigator toolbar.
Review in the list the status of your Load Plan run.
Double-click this Load Plan run to open the Load Plan Run editor.
In the Load Plan Run editor, select the Steps tab.
Review the state of the Load Plan steps. On this tab, you can perform the following tasks:
- Click Refresh in the Editor toolbar to update the content of the table.
- For the Run Scenario steps, you can click in the Session ID column to open the session started by this Load Plan for this step.

21.2.3 Handling Failed Sessions

When your session ends in error or with a warning, you can analyze the error in Operator Navigator.

To analyze an error:

In the Operator Navigator, identify the session, the step and the task in error.
Double click the task in error. The Task editor opens.
On the Definition tab in the Execution Statistics section, the return code and message give the error that stopped the session.
On the Code tab, the source and target code for the task is displayed and can be reviewed and edited.
On the Connection tab, you can review the source and target connections against which the code is executed.

You can fix the code of the command in the Code tab and apply your changes. Restarting a Session is possible after performing this action. The session will restart from the task in error.

Note:

Fixing the code in the session's task does not fix the source object that was executed (interface, procedure, package or scenario). This source object must be fixed in Designer Navigator and the scenario (if any) must be regenerated. Modifying the code within the session is useful for debugging issues.

WARNING:

When a session fails, all connections and transactions to the source and target systems are rolled back. As a consequence, uncommitted statements on transactions are not applied.

21.2.4 Reviewing Successful Sessions

When your session ends successfully, you can view the changes performed in Operator Navigator. These changes include record statistics such as the number of inserts, updates, deletes, errors, and the total number of rows as well as execution statistics indicating start and end time of the execution, the duration in seconds, the return code, and the message (if any).

Session level statistics aggregate the statistics of all the steps of this session, and each step's statistics aggregate the statistics of all the tasks within this step.

To review the execution statistics:

In the Operator Navigator, identify the session, the step or the task to review.
Double click the session, the step or the task. The corresponding editor opens.
The record and execution statistics are displayed on the Definition tab. Note that for session steps in which an interface has been executed or a datastore check has been performed also the target table details are displayed.

Record Statistics

Properties	Description
No. of Inserts	Number of rows inserted during the session/step/task.
No. of Updates	Number of rows updated during the session/step/task.
No. of Deletes	Number of rows deleted during the session/step/task.
No. of Errors	Number of rows in error in the session/step/task.
No. of Rows	Total number of rows handled during this session/step/task.

Execution Statistics

Properties	Description
Start	Start date and time of execution of the session/step/task.
End	End date and time of execution of the session/step/task.
Duration (seconds)	The time taken for execution of the session/step/task.
Return code	Return code for the session/step/task.

Target Table Details

Properties	Description
Table Name	Name of the target datastore.
Model Code	Code of the Model in which the target datastore is stored.
Resource Name	Resource name of the target datastore.
Logical Schema	Logical schema of this datastore.
Forced Context Code	The context of the target datastore.

21.2.5 Handling Failed Load Plans

When a Load Plan ends in error, review the sessions that have failed and caused the Load Plan to fail. Fix the source of the session failure.

You can restart the Load Plan instance. See Section 20.7, "Restarting a Load Plan Run" for more information.

Note that it will restart depending on the Restart Type defined on its steps. See Section 14.2.4, "Handling Load Plan Exceptions and Restartability" for more information.

21.2.6 Reviewing Successful Load Plans

When your Load Plan ends successfully, you can review the execution statistics from the Load Plan run editor.

You can also review the statistics for each session started for this Load Plan in the session editor.

To review the Load Plan run execution statistics:

In the Operator Navigator, identify the Load Plan run to review.
Double click the Load Plan run. The corresponding editor opens.
The record and execution statistics are displayed on the Steps tab.

21.3 Managing your Executions

Managing your development executions takes place in Operator Navigator. You can manage your executions during the execution process itself or once the execution has finished depending on the action that you wish to perform. The actions that you can perform are:

Managing Sessions
Managing Load Plan Executions
Managing the Log
Managing Scenarios and Load Plans
Managing Schedules

21.3.1 Managing Sessions

Managing sessions involves the following tasks

New sessions can be created by executing run-time objects or scenarios. See Chapter 20, "Running Integration Processes" for more information on starting sessions.
Sessions in progress can be aborted. How to stop sessions is covered in Section 20.5, "Stopping a Session".
Sessions failed, or stopped by user action can be restarted. Restarting sessions is covered in Section 20.4, "Restarting a Session".

In addition to these tasks, it may be necessary in production to deal with stale sessions.

21.3.1.1 Cleaning Stale Sessions

Stale sessions are sessions that are incorrectly left in a running state after an agent or repository crash.

The Agent that started a session automatically detects when this session becomes stale and changes it to Error status. You can manually request specific Agents to clean stale sessions in Operator Navigator or Topology Navigator.

To clean stale sessions manually:

Do one of the following:
- From the Operator Navigator toolbar menu, select Clean Stale Sessions.
- In Topology Navigator, from the Physical Architecture accordion, select an Agent, right-click and select Clean Stale Sessions.
The Clean Stale Sessions Dialog opens.
In the Clean Stale Sessions Dialog specify the criteria for cleaning stale sessions:
- From the list, select the Agents that will clean their stale sessions.
  
  Select Clean all Agents if you want all Agents to clean their stale sessions.
- From the list, select the Work Repositories you want to clean.
  
  Select Clean all Work Repositories if you want to clean stale sessions in all Work Repositories.
Click Clean to start the cleaning process.

21.3.2 Managing Load Plan Executions

Managing Load Plan Executions involves the following tasks:

New Load Plan Instances and Runs can be created by executing Load Plans. See Section 20.6, "Executing a Load Plan" for more information on starting Load Plans.
Load Plan Runs in progress can be aborted. How to stop Load Plan runs is covered in Section 20.8, "Stopping a Load Plan Run".
Load Plan Runs failed, or stopped by user action can be restarted. Restarting Load Plan Runs is covered in Section 20.7, "Restarting a Load Plan Run".

21.3.3 Managing the Log

Oracle Data Integrator provides several solutions for managing your log data:

Filtering Sessions to display only certain execution sessions in Operator Navigator
Purging the Log to remove the information of past sessions
Organizing the Log with Session Folders
Exporting and Importing Log Data for archiving purposes

21.3.3.1 Filtering Sessions

Filtering log sessions allows you to display only certain sessions in Operator Navigator, by filtering on parameters such as the user, status or duration of sessions. Sessions that do not meet the current filter are hidden from view, but they are not removed from the log.

To filter out sessions:

In the Operator Navigator toolbar menu, click Filter. The Define Filter editor opens.
In the Define Filter Editor, set the filter criteria according to your needs. Note that the default settings select all sessions.
- Session Number: Use blank to show all sessions.
- Session Name: Use % as a wildcard. For example DWH% matches any session whose name begins with DWH.
- Session's execution Context
- Agent used to execute the session
- User who launched the session
- Status: Running, Waiting etc.
- Date of execution: Specify either a date From or a date To, or both.
- Duration greater than a specified number of seconds
Click Apply for a preview of the current filter.
Click OK.

Sessions that do not match these criteria are hidden in the Session List accordion. The Filter button on the toolbar is activated.

To deactivate the filter click Filter in the Operator toolbar menu. The current filter is deactivated, and all sessions appear in the list.

21.3.3.2 Purging the Log

Purging the log allows you to remove past sessions and Load Plan runs from the log. This procedure is used to keeping a reasonable volume of sessions and Load Plans archived in the work repository. It is advised to perform a purge regularly. This purge can be automated using the OdiPurgeLog tool in a scenario.

To purge the log:

From the Operator Navigator toolbar menu select Connect Navigator > Purge Log... The Purge Log editor opens.

In the Purge Log editor, set the criteria listed in Table 21-3 for the sessions or Load Plan runs you want to delete.

Table 21-3 Purge Log Parameters

Parameter	Description
Purge Type	Select the objects to purge.
From ... To	Sessions and/or Load Plan runs in this time range will be deleted. When you choose to purge session logs only, then the sessions launched as part of the Load Plan runs are not purged even if they match the filter criteria.When you purge Load Plan runs, the Load Plan run which matched the filter criteria and the sessions launched directly as part of the Load Plan run and its child/grand sessions will be deleted.
Context	Sessions and/or Load Plan runs executed in this context will be deleted.
Agent	Sessions and/or Load Plan runs executed by this agent will be deleted.
Status	Session and/or Load Plan runs in this status will be deleted.
User	Sessions and/or Load Plan runs executed by this user will be deleted.
Name	Sessions and/or Load Plan runs matching this session name will be deleted. Note that you can specify session name masks using % as a wildcard.
Purge scenario reports	If you select Purge scenario reports, the scenario reports (appearing under the execution node of each scenario) will also be purged.

Only the sessions and/or Load Plan runs matching the specified filters will be removed:

When you choose to purge session logs only, then the sessions launched as part of the Load Plan runs are not purged even if they match the filter criteria.
When you purge Load Plan runs, the Load Plan run which matched the filter criteria and the sessions launched directly as part of Load Plan run and its child/grand sessions will be deleted.
When aLoad Plan run matches the filter, all its attached sessions are also purged irrespective of whether they match the filter criteria or not.

Click OK.

Oracle Data Integrator removes the sessions and/or Load Plan runs from the log.

Note:

It is also possible to delete sessions or Load Plan runs by selecting one or more sessions or Load Plan runs in Operator Navigator and pressing the Delete key. Deleting a Load Plan run in this way, deletes the corresponding sessions.

21.3.3.3 Organizing the Log with Session Folders

You can use session folders to organize the log. Session folders automatically group sessions and Load Plan Runs that were launched with certain keywords. Session folders are created under the Keywords node on the Session List or Load Plan Executions accordions.

Each session folder has one or more keywords associated with it. Any session launched with all the keywords of a session folder is automatically categorized beneath it.

To create a new session folder:

In Operator Navigator, go to the Session List or Load Plan Executions accordion.
Right-click the Keywords node and select New Session Folder.
Specify a Folder Name.
Click Add to add a keyword to the list. Repeat this step for every keyword you wish to add.

Note:

Only sessions or load plans with all the keywords of a given session folder will be shown below that session folder. Keyword matching is case sensitive.

Table 21-4 lists examples of how session folder keywords are matched.

Table 21-4 Matching of Session Folder Keywords

Session folder keywords	Session keywords	Matches?
DWH, Test, Batch	Batch	No - all keywords must be matched.
Batch	DWH, Batch	Yes - extra keywords on the session are ignored.
DWH, Test	Test, dwh	No - matching is case-sensitive.

To launch a session with keywords, you can for example start a scenario from a command line with the -KEYWORDS parameter. Refer to Chapter 20, "Running Integration Processes" for more information.

Note:

Session folder keyword matching is dynamic. If the keywords for a session folder are changed or if a new folder is created, existing sessions are immediately re-categorized.

21.3.3.4 Exporting and Importing Log Data

Export and import log data for archiving purposes.

Exporting Log Data

Exporting log data allows you to export log files for archiving purposes.

To export the log:

From the Operator Navigator toolbar menu select Connect Navigator > Export > Log

In the Export the log dialog, set the log export parameters as described in Table 21-5.

Table 21-5 Log Export Parameters

Properties	Description
Export to directory	Directory in which the export file will be created.
Export to zip file	If this option is selected, a unique compressed file containing all log export files will be created. Otherwise, a set of log export files is created.
Zip File Name	Name given to the compressed export file.
Filters	This set of options allow to filter the log files to export according to the specified parameters.
From / To	Date of execution: specify either a date From or a date To, or both.
Agent	Agent used to execute the session. Leave the default All Agents value, if you do not want to filter based on a given agent.
Context	Session's execution Context. Leave the default All Contexts value, if you do not want to filter based on a context.
Status	The possible states are `Done`, `Error`, `Queued`, `Running`, `Waiting`, `Warning` and `All States`. Leave the default All States value, if you do not want to filter based on a given session state.
User	User who launched the session. Leave the default All Users value, if you do not want to filter based on a given user.
Session Name	Use `%` as a wildcard. For example `DWH%` matches any session whose name begins with `DWH`.
Advanced options	This set of options allow to parameterize the output file format.
Character Set	Encoding specified in the export file. Parameter encoding in the XML file header. `<?xml version="1.0"` `encoding="ISO-8859-1"?>`
Java Character Set	Java character set used to generate the file.

Click OK.

The log data is exported into the specified location.

Note that you can also automate the log data export using the OdiExportLog tool.

Importing Log Data

Importing log data allows you to import into your work repository log files that have been exported for archiving purposes.

To import the log:

From the Operator Navigator toolbar menu select Connect Navigator > Import > Log
In the Import of the log dialog:
1. Select the Import Mode. Note that sessions can only be imported in Synonym Mode INSERT mode. Refer to Section 19.1.3, "Import Modes" for more information.
2. Select whether you want to import the files From a Folder or From a ZIP file.
3. Enter the file import folder or zip file.
4. Click OK.

The specified folder or ZIP file is imported into the work repository.

21.3.4 Managing Scenarios and Load Plans

You can also manage your executions in Operator Navigator by using scenarios or Load Plans.

Before running a scenario, you need to generate it in Designer Navigator or import from a file. See Chapter 13, "Working with Scenarios". Load Plans are also created using Designer Navigator, but can also be modified using Operator Navigator. See Chapter 14, "Working with Load Plans" for more information.

Launching a scenario from Operator Navigator is covered in Section 20.3.1, "Executing a Scenario from ODI Studio" and how to run a Load Plan is described in Section 20.6, "Executing a Load Plan".

21.3.4.1 Load Plan and Scenario Folders

In Operator Navigator, scenarios and Load Plans can be grouped into Load Plan and Scenario folders to facilitate organization. Load Plan and Scenario folders can contain other Load Plan and Scenario folders.

To create a Load Plan and Scenario folder:

In Operator Navigator go to the Load Plans and Scenarios accordion.
From the Load Plans and Scenarios toolbar menu, select New Load Plan and Scenario Folder.
On the Definition tab of the Load Plan and Scenario Folder editor enter a name for your folder.
From the File menu, select Save.

You can now reorganize your scenarios and Load Plans. Drag and drop them into the Load Plan and Scenario folder.

21.3.4.2 Importing Load Plans, Scenarios, and Solutions in Production

A Load Plan or a scenario generated from Designer can be exported and then imported into a development or execution repository. This operation is used to deploy Load Plans and scenarios in a different repository, possibly in a different environment or site.

Importing a Load Plan or scenario in a development repository is performed via Designer or Operator Navigator. With a execution repository, only Operator Navigator is available for this purpose.

See Section 13.6, "Importing Scenarios in Production" for more information on how to import a scenario in production and Section 14.4.3.2, "Importing Load Plans" for more information on the Load Plan import.

Similarly, a solution containing several scenarios can be imported to easily transfer and restore a group of scenarios at once. See Chapter 18, "Working with Version Management" for more information. Note that when connected to an execution repository, only scenarios may be restored from solutions.

21.3.5 Managing Schedules

A schedule is always attached to one scenario or one Load Plan. Schedules can be created in Operator Navigator. See Section 20.9, "Scheduling Scenarios and Load Plans" for more information.

You can also import an already existing schedule along with a scenario or Load Plan import. See Section 13.6, "Importing Scenarios in Production" and Section 14.4.3, "Exporting, Importing and Versioning Load Plans"for more information.

You can view the scheduled tasks of all your agents or you can view the scheduled tasks of one particular agent. See Section 20.9.1.3, "Displaying the Schedule" for more information.