Before getting started with the Python SDK, it is important to understand the basic concepts of Data Transforms.

Deployment

Represents an instance where Data Transforms is running.

Data Transforms can be deployed from three places.

• OCI Marketplace
• ADB-S
• OCI GoldenGate Service.

Accessing the deployment varies based on the type of deployment. Data Transforms Python SDK provides unified access for all of these deployment types.

Deployment Config

Includes the configuration parameters for accessing the Deployment. The Deployment Config leverages Python’s configparser with customised parameter(s) for Data Transforms.

• Structure of the deployment configuration

  • The deployment.config file is where all the deployments are configured (default implementation). This file captures one or more deployments with their connection parameters.

    • Every deployment is identified by a unique name, represented as [name_of_the_deployment].

    • The deployment file must have one ACTIVE deployment, identified with [ACTIVE] entry. If deployment.config does not have any active configuration available, the API execution(s) will be skipped.

    • Active Deployment is identified by a unique name as below:

      [ACTIVE]
      deployment=<deployment-name>

  • Marketplace based deployment:

    [name_of_deployment]
    data_transforms_ip=<ip-address>
    data_transforms_user=<user_account>

    where

    • <ip-address> is obtained from Public IP of the compute instance where Data Transforms was provisioned from OCI Marketplace.
    • data_forms_user is the login user created for accessing Data Transforms.
  • Autonomous Database based deployment:

    [name_of_deployment]
    data_transforms_url=<data-transforms-url>
    data_transforms_user=<user>
    tenancy_ocid=<tenancy_ocid>
    adw_name=<name of the ADW instance>
    adw_ocid=<ADW ocid>

    where

    • <data-transforms-url> is the URL of your Data Transforms instance.
      To get the URL:

      1. From the Oracle Cloud Infrastructure left navigation menu click Oracle Database, select Autonomous Database, select the Autonomous Database instance from the list, and navigate to the Autonomous Database details page.
      2. Select the Tool configuration tab.
      3. Copy the URL from the Public access URL text box excluding the trailing slash (/).

         For example, https://example.adb.us-phoenix-1.oraclecloudapps.com/odi

    • <user> is the login user created for accessing Data Transforms.
    • <tenancy_ocid> is the tenancy OCID from the OCI Console.
    • <name of the ADW instance> is the name of the Autonomous Database instance where the compute node will be built.

      You will find the database name under General Information in the Autonomous Database details page.

    • <ADW ocid> is the OCID of the Autonomous Database.
• OCI GoldenGate based deployments:

  [name_of_deployment]
  data_transforms_url=<data-transforms-url>
  data_transforms_user=<user>

• Sample Deployment Config file:

  [ACTIVE]
  deployment=mydev_env
  
  [my_dev_env]
  data_transforms_ip=127.0.0.1
  data_transforms_user=user
  
  [abds_env]
  data_transforms_url=https://my-adbs-transforms-url
  data_transforms_user=user
  tenancy_ocid=ocid1.tenancy.oc1..tenancy_ocid
  adw_name=mydw
  adw_ocid=ocid1.autonomousdatabase.oc1.phx.adw_ocid
  
  [qa_env]
  data_transforms_url=https://my-qa-env-url
  data_transforms_user=user
  
  [scott_env]
  data_transforms_ip=127.1.0.1
  data_transforms_user=scott_env_user

Workbench

A dedicated workspace for the developer. A directory where all the Python scripts are maintained. The workbench usually has the following:

• A deployment.config file.
• Python scripts for connections, data entities, projects.
• Artifacts in the respective project folder.

Artifact Resolution and Create-or-Update Behavior

Data Transforms artifacts—including connections, data entities, data loads, data flows, workflows, and schedules—are identified by a unique name.

Before persisting an artifact, the SDK resolves that name against the active Data Transforms deployment. This provides automatic create-or-update behavior, so developers do not need to look up the artifact or choose between create and update operations themselves.

• If an artifact with the specified name already exists, the SDK updates the existing artifact.
• If no artifact with that name exists, the SDK creates a new artifact.

For example, workbench.save_connection(connection_obj) checks for an existing connection with the supplied name. When a matching connection is found, the SDK updates it; otherwise, it creates a new connection.