Transformation Configuration Templates

This topic explains the configuration parameters for the different transformations supported by the Oracle NoSQL Database Migrator. For the complete configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

Oracle NoSQL Database Migrator lets you modify the data, that is, add data transformations as part of the migration activity. You can define multiple transformations in a single migration. In such a case, the order of transformations is vital because the source data undergoes each transformation in the given order. The output of one transformation becomes the input to the next one in the migrator pipeline.

The different transformations supported by the NoSQL Data Migrator are:

Table 1-2 Transformations

Transformation Config Attribute	You can use this transformation to ...
`ignoreFields`	Ignore the identified columns from the source row before writing to the sink.
`includeFields`	Include the identified columns from the source row before writing to the sink.
`renameFields`	Rename the identified columns from the source row before writing to the sink.
`aggregateFields`	Aggregate multiple columns from the source into a single column in the sink. As part of this transformation, you can also identify the columns that you want to exclude in the aggregation. Those fields will be skipped from the aggregated column.

You can find the configuration template for each supported transformation below.

ignoreFields

The configuration file format for the ignoreFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "ignoreFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

ignoreFields

Purpose: An array of the column names to be ignored from the source records.

Note:
You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
Data Type: array of strings
Mandatory (Y/N): Y
Example: To ignore the columns named "name" and "address" from the source record:

"ignoreFields" : ["name","address"]

includeFields

The configuration file format for the includeFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "includeFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

includeFields

Purpose: An array of the column names to be included from the source records. It only includes the fields specified in the array, the rest of the fields are ignored.

Note:
The NoSQL Database Migrator tool throws an error if you specify an empty array. Additionally, you can specify only the top-level fields. The NoSQL Database Migrator tool does not apply transformations to the data in the nested fields.
Data Type: array of strings
Mandatory (Y/N): Y
Example: To include the columns named "age" and "gender" from the source record:

"includeFields" : ["age","gender"]

renameFields

The configuration file format for the renameFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "renameFields" : {
    "<old_name>" : "<new_name>",
    "<old_name>" : "<new_name>,"
    .....
  }
}

Transformation Parameter

renameFields

Purpose: Key-Value pairs of the old and new names of the columns to be renamed.

Note:
You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
Data Type: JSON object
Mandatory (Y/N): Y
Example: To rename the column named "residence" to "address" and the column named "_id" to "id":

"renameFields" : { "residence" : "address", "_id" : "id" }

aggregateFields

The configuration file format for the aggregateFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "aggregateFields" : {
    "fieldName" : "name of the new aggregate field",
    "skipFields" : ["<field1>","<field2">,...]
  }
}

Transformation Parameter

aggregateFields

Purpose: Name of the aggregated field in the sink.
Data Type: string
Mandatory (Y/N): Y

Example: If the given record is:

{
  "id" : 100,
  "name" : "john",
  "address" : "USA",
  "age" : 20
}

If the aggregate transformation is:

"aggregateFields" : {
  "fieldName" : "document",
  "skipFields" : ["id"]
}

The aggregated column in the sink looks like:

{
  "id": 100,
  "document": {
    "name": "john",
    "address": "USA",
    "age": 20
  }
}