You can migrate a JSON source file by specifying the file path or a directory in the source configuration template.

{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-02T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-04T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Source Configuration Template

"source": {
  "type": "file",
  "format": "json",
  "dataPath": "<path/to/JSON/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "<path/to/schema/file>"
  }
},

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "json"

Unique Configuration Parameters

• dataPath
• schemaInfo
• schemaInfo.schemaPath

dataPath

• Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

  You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  • Specifying a JSON file

    "dataPath" : "/home/user/sample.json"

  • Specifying a directory

    "dataPath" : "/home/user"

schemaInfo

• Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

• Data Type: Object
• Mandatory (Y/N): N

schemaInfo.schemaPath

• Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  "schemaInfo": {
    "schemaPath": "<path to the schema file>"
  }

You can migrate a JSON file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-04T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>
}

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "object_storage_oci"

• format

  Use "format" : "json"

• endpoint
  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• namespace

  Example: "namespace" : "my-namespace"

• bucket

  Example: "bucket" : "my-bucket"

• prefix
  Example:

  1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
  2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
• credentials
  Example:

  1. "credentials" : "/home/user/.oci/config"
  2. "credentials" : "/home/user/security/config"
• credentialsProfile
  Example:

  1. "credentialsProfile" : "DEFAULT"
  2. "credentialsProfile" : "ADMIN_USER"
• useInstancePrincipal

  Example: "useInstancePrincipal" : true

• useDelegationToken

  Example: "useDelegationToken" : true

  Note:

  The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

Unique Configuration Parameters

• schemaInfo
• schemaInfo.schemaObject

schemaInfo

• Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

• Data Type: Object
• Mandatory (Y/N): N

schemaInfo.schemaObject

• Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  "schemaInfo": {
    "schemaObject": "mytable/Schema/schema.ddl"
  },

You can migrate a MongoDB exported JSON data by specifying the file or directory in the source configuration template.

MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either Canonical or Relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

For more information on the MongoDB Extended JSON (v2) file, See mongoexport_formats.

For more information on the generation of MongoDB-formatted JSON file, see mongoexport for more information.

{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Source Configuration Template

"source": {
  "type": "file",
  "format": "mongodb_json",
  "dataPath": "</path/to/json/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "</path/to/schema/file>"
  }
}

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "mongodb_json"

Unique Configuration Parameters

• dataPath
• schemaInfo
• schemaInfo.schemaPath

dataPath

• Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

  You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool.

  If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  • Specifying a MongoDB formatted JSON file

    "dataPath" : "/home/user/sample.json"

  • Specifying a directory

    "dataPath" : "/home/user"

schemaInfo

• Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

• Data Type: Object
• Mandatory (Y/N): N

schemaInfo.schemaPath

• Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  "schemaInfo" : {
    "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
  }

You can migrate the MongoDB exported JSON data in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information. MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. Both formats are supported in the OCI Object Storage bucket.

{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "object_storage_oci"

• format

  Use "format" : "mongodb_json"

• endpoint
  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• namespace

  Example: "namespace" : "my-namespace"

• bucket

  Example: "bucket" : "my-bucket"

• prefix
  Example:

  1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
  2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)

  Note:

  If you do not provide any value, all the objects present in the bucket are migrated.
• credentials
  Example:

  1. "credentials" : "/home/user/.oci/config"
  2. "credentials" : "/home/user/security/config"
• credentialsProfile
  Example:

  1. "credentialsProfile" : "DEFAULT"
  2. "credentialsProfile" : "ADMIN_USER"
• useInstancePrincipal

  Example: "useInstancePrincipal" : true

Unique Configuration Parameters

• schemaInfo
• schemaInfo.schemaObject

schemaInfo

• Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

• Data Type: Object
• Mandatory (Y/N): N

schemaInfo.schemaObject

• Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  "schemaInfo": {
    "schemaObject": "mytable/Schema/schema.ddl"
  }

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "aws_s3"

• format

  Use "format" : "dynamodb_json"

  Note:

  If the value of the type parameter is aws_s3, then the format must be dynamodb_json.

Unique Configuration Parameters

• s3URL
• credentials
• credentialsProfile
• ttlAttributeName

s3URL

• Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. The valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The NoSQL Database Migrator will look for json.gz files in the prefix during import.

  Note:

  You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
• Data Type: string
• Mandatory (Y/N): Y
• Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be

credentials

• Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. For more details on the credentials file, see Configuration and credential file settings .
• Data Type: string
• Mandatory (Y/N): N
• Example:

  "credentials" : "/home/user/.aws/credentials"

  "credentials" : "/home/user/security/credentials

  Note:

  The NoSQL Database Migrator does not log any of the credentials information. You must properly protect the credentials file from unauthorized access.

credentialsProfile

• Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, NoSQL Database Migrator uses the default profile. For more details on the credentials file, see Configuration and credential file settings .
• Data Type: string
• Mandatory (Y/N): N
• Example:

  "credentialsProfile" : "default"

  "credentialsProfile" : "test"

ttlAttributeName

• Purpose: Specifies the name of TTL attribute present in the exported DynamoDB table data. You include this parameter only if the DynamoDB table data has a TTL attribute and you want to set the TTL value on imported data while importing to NoSQL Database.

  Note:

  To import with the TTL metadata, you must set the includeTTL configuration parameter to true in the sink configuration template (nosqldb and nosqldb_cloud).
• Data Type: string
• Mandatory (Y/N): N
• Example: "ttlAttributeName" : "ttl"

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "dynamodb_json"

Unique Configuration Parameter

• dataPath
• ttlAttributeName

dataPath

• Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You must copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the data sub-directory.
• Data Type: string
• Mandatory (Y/N): Y
• Example:
  • Specifying a file

    "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"

  • Specifying a directory

    "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"

ttlAttributeName

• Purpose: Specifies the name of TTL attribute present in the exported DynamoDB table data. You include this parameter only if the DynamoDB table data has a TTL attribute and you want to set the TTL value on imported data while importing to NoSQL Database.

  Note:

  To import with the TTL metadata, you must set the includeTTL configuration parameter to true in the sink configuration template (nosqldb and nosqldb_cloud).
• Data Type: string
• Mandatory (Y/N): N
• Example: "ttlAttributeName" : "ttl"

You can migrate a table from Oracle NoSQL Database by specifying the table name in the source configuration template.

{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type": "nosqldb",
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "table" : "<fully qualified table name>", 
  "includeTTL": <true|false>,    
  "security" : "</path/to/store/security/file>",
  "requestTimeoutMs" : 5000
}

Source Parameters

Common Configuration Parameter

• type

  Use "type" : "nosqldb"

• security

  Example:

  "security" : "/home/user/client.credentials"

  Example security file content for password file based authentication:

  oracle.kv.password.noPrompt=true
  oracle.kv.auth.username=admin
  oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
  oracle.kv.transport=ssl
  oracle.kv.ssl.trustStore=/home/nosql/client.trust
  oracle.kv.ssl.protocols=TLSv1.2
  oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

  Example security file content for wallet based authentication:

  oracle.kv.password.noPrompt=true
  oracle.kv.auth.username=admin
  oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
  oracle.kv.transport=ssl
  oracle.kv.ssl.trustStore=/home/nosql/client.trust
  oracle.kv.ssl.protocols=TLSv1.2
  oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

• requestTimeoutMs

  Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

• storeName
• helperHosts
• table
• includeTTL

storeName

• Purpose: Name of the Oracle NoSQL Database store.

• Data Type: string
• Mandatory (Y/N): Y
• Example: "storeName" : "kvstore"

helperHosts

• Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

• Data Type: array of strings
• Mandatory (Y/N): Y
• Example: "helperHosts" : ["localhost:5000","localhost:6000"]

table

• Purpose: Fully qualified table name from which to migrate the data.

  Format: [namespace_name:]<table_name>

  If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  • With the DEFAULT namespace "table" :"mytable"

  • With a non-default namespace "table" : "mynamespace:mytable"

  • To specify a child table "table" : "mytable.child"

includeTTL

• Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

  If you do not specify this parameter, it defaults to false.

  Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:

  //ROW1
  {
    "id" : 1,
    "name" : "abc",
    "_metadata" : {
      "expiration" : 1634601600000
    }
  }
  
  //ROW2
  {
    "id" : 2,
    "name" : "xyz"
  }

• Data Type: boolean
• Mandatory (Y/N): N
• Example: "includeTTL" : true

You can migrate a table from Oracle NoSQL Database Cloud Service by specifying the name or OCID of the compartment in which the table resides in the source configuration template.

{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service endpoint URL or region ID>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "<path/to/oci/credential/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "readUnitsPercent" : <table readunits percent>,
  "includeTTL": <true|false>,
  "requestTimeoutMs" : <timeout in milli seconds>
}

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "nosqldb_cloud"

• endpoint
  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• credentials
  Example:

  1. "credentials" : "/home/user/.oci/config"
  2. "credentials" : "/home/user/security/config"
• credentialsProfile
  Example:

  1. "credentialsProfile" : "DEFAULT"
  2. "credentialsProfile" : "ADMIN_USER"
• useInstancePrincipal

  Example: "useInstancePrincipal" : true

• useDelegationToken

  Example: "useDelegationToken" : true

  Note:

  The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.
• requestTimeoutMs

  Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

• table
• compartment
• readUnitsPercent
• includeTTL

table

• Purpose: Name of the table from which to migrate the data.

• Data Type: string
• Mandatory (Y/N): Y
• Example:
  • To specify a table "table" : "myTable"
  • To specify a child table "table" : "mytable.child"

compartment

• Purpose: Specifies the name or OCID of the compartment in which the table resides.

  If you do not provide any value, it defaults to the root compartment.

  You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

• Data Type: string
• Mandatory (Y/N): Y, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

  Note:

  If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
• Example:

  • Compartment name

    "compartment" : "mycompartment"

  • Compartment name qualified with its parent compartment

    "compartment" : "parent.childcompartment"

  • No value provided. Defaults to the root compartment.

    "compartment": ""

  • Compartment OCID

    "compartment" : "ocid1.tenancy.oc1...4ksd"

readUnitsPercent

• Purpose: Percentage of table read units to be used while migrating the NoSQL table.

  The default value is 90. The valid range is any integer between 1 to 100. The amount of time required to migrate data is directly proportional to this attribute. It is better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

  To learn the daily limits on throughput changes, see Cloud Limits in the Oracle NoSQL Database Cloud Service document.

  See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

• Data Type: integer
• Mandatory (Y/N): N
• Example: "readUnitsPercent" : 90

includeTTL

• Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

  If you do not specify this parameter, it defaults to false.

  Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:

  //ROW1
  {
    "id" : 1,
    "name" : "abc",
    "_metadata" : {
      "expiration" : 1634601600000
    }
  }
  
  //ROW2
  {
    "id" : 2,
    "name" : "xyz"
  }

• Data Type: boolean
• Mandatory (Y/N): N
• Example: "includeTTL" : true

You can migrate a CSV file or a directory containing the CSV data by specifying the file name or directory in the source configuration template.

1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Source Configuration Template

"source" : {
  "type" : "file",
  "format" : "csv",
  "dataPath": "</path/to/a/csv/[file|dir]>",
  "hasHeader" : <true | false>,
  "columns" : ["column1", "column2", ....],
  "csvOptions": {
    "encoding": "<character set encoding>",
    "trim": "<true | false>"
 }
}

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "csv"

Unique Configuration Parameters

• dataPath

• hasHeader

• columns

• csvOptions

• csvOptions.encoding

• csvOptions.trim

datapath

• Purpose: Specifies the absolute path to a file or directory containing the CSV data for migration. If you specify a directory, NoSQL Database Migrator imports all the files with the .csv or .CSV extension in that directory. All the CSV files are copied into a single table, but not in any particular order.

  CSV files must conform to the RFC4180 standard. You must ensure that the data in each CSV file matches with the NoSQL Database table schema defined in the sink table. Sub-directories are not supported.

• Data Type: string
• Mandatory (Y/N): Y
• Example:

  • Specifying a CSV file

    "dataPath" : "/home/user/sample.csv"

  • Specifying a directory

    "dataPath" : "/home/user"

hasHeader

• Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

• Data Type: Boolean
• Mandatory (Y/N): N

• Example: "hasHeader" : "false"

columns

• Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.

  Note:

  • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
  • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
  • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
• Data Type: Array of Strings
• Mandatory (Y/N): N

• Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

• Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

• Data Type: Object
• Mandatory (Y/N): N

csvOptions.encoding

• Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

• Data Type: String
• Mandatory (Y/N): N

• Example: "encoding" : "UTF-8"

csvOptions.trim

• Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

• Data Type: Boolean
• Mandatory (Y/N): N

• Example: "trim" : "true"

You can migrate a CSV file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "csv",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
   "hasHeader" : <true | false>,
   "columns" : ["column1", "column2", ....],
   "csvOptions" : {         
     "encoding" : "<character set encoding>",
     "trim" : <true | false>
   }
 }

Source Parameters

Common Configuration Parameters

• type

  Use "type" : "object_storage_oci"

• format

  Use "format" : "csv"

• endpoint
  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• namespace

  Example: "namespace" : "my-namespace"

• bucket

  Example: "bucket" : "my-bucket"

  Note:

  • The NoSQL Database Migrator imports all the files with the .csv or .CSV extension object-wise and copies them into a single table in the same order.

  • The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator.

• prefix
  Example:

  1. "prefix" : "my_table/Data/000000.csv" (migrates only 000000.csv)
  2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
• credentials
  Example:

  1. "credentials" : "/home/user/.oci/config"
  2. "credentials" : "/home/user/security/config"
• credentialsProfile
  Example:

  1. "credentialsProfile" : "DEFAULT"
  2. "credentialsProfile" : "ADMIN_USER"
• useInstancePrincipal

  Example: "useInstancePrincipal" : true

Unique Configuration Parameters

• hasHeader
• columns
• csvOptions
• csvOptions.encoding
• csvOptions.trim

hasHeader

• Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

• Data Type: Boolean
• Mandatory (Y/N): N

• Example: "hasHeader" : "false"

columns

• Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.

  Note:

  • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
  • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
  • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
• Data Type: Array of Strings
• Mandatory (Y/N): N

• Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

• Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

• Data Type: Object
• Mandatory (Y/N): N

csvOptions.encoding

• Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

• Data Type: String
• Mandatory (Y/N): N

• Example: "encoding" : "UTF-8"

csvOptions.trim

• Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

• Data Type: Boolean
• Mandatory (Y/N): N

• Example: "trim" : "true"