The ABS Event Handler name is pre-set to the value abs and must be configured to match your ADLS Gen2 configuration.

The following is an example of editing a property of the S3 Event Handler: gg.eventhandler.abs.bucketMappingTemplate=container1

For more information about integrating with ADLS Gen2, see Azure Blob Storage Configuration.

For more information about integrating with s3, see S3 Event handler configuration.

The GCS Event Handler name is pre-set to the value gcs and must be configured to match your GCS configuration. The following is an example of editing a GCS Event Handler property: gg.eventhandler.gcs.bucketMappingTemplate=bucket1

The following is an example of editing a property of the S3 Event Handler: gg.eventhandler.abs.bucketMappingTemplate=container1

For more information about integrating with GCS, see GCS Event Handler Configuration.

The Databricks Event Handler name is pre-set to the value databricks.

The following are the configuration properties available for the Databricks Event handler, the required ones must be changed to match your Databricks configuration.

The driver requires the following compute resource configuration settings:

Oracle GoldenGate replicates data to Databricks managed tables. Managed tables use the DELTA table format.

External tables are file-backed tables that reference data stored in an external location.

External location is an object storage such as ADLS Gen2, AWS S3, or GCS.

A storage credential represents an authentication and authorization mechanism for accessing data stored on your cloud tenant.

To create an external location:

• Log in to your Databricks workspace.
• In the sidebar, click Catalog.
• Click External Data, then click on External Locations.
• Click Create Location.
• In the Create a new external location form, enter the following details:
  • External location name: Enter a name for the external location.
  • Storage credential: Choose the storage credential that you created earlier from the drop-down menu.
  • URL: Enter the bucket path that you want to use as the external location. For example:

    abfss://[email protected]/

  • Click Advanced Options, check the box Limit to read-only use.
  • Click Create.

To create an external location:

• Log in to your Databricks workspace.
• In the sidebar, click Catalog.
• Click External Data, and then click External Locations.
• Click Create Location.
• In the Create a new external location form, enter the following details:
  • External location name: Enter a name for the external location.
  • Storage credential: Choose the storage credential that you created earlier from the drop-down menu.
  • URL: Enter the bucket path that you want to use as the external location. For example:

    gs://gcs-bucket/ogg

  • Click Advanced Options, check the box Limit to read-only use.
  • Click Create.

Access to ADLS Gen2 can be configured using the storage account name and key.

Storage account key needs can be securely stored in Databricks secrets.

fs.azure.account.key.storageaccountname.dfs.core.windows.net {{secrets/gg/azureAccountKey-for-storageaccountname}}

In this example, storageaccountname is the Azure storage account name, and {{secrets/gg/azureAccountKey-for-storageaccountname}} is the Databricks secret that contains the storage account key.

Access to Google Cloud Storage can be configured using a Google service account that has permission to access the storage bucket.

Access to AWS S3 can be configured in multiple ways. The following environment variables can be set in the Spark configuration to access S3:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

AWS_SECRET_ACCESS_KEY={{secrets/gg/aws_secret_access_key}}
AWS_ACCESS_KEY_ID={{secrets/gg/aws_access_key_id}}

To create a Databricks secret:

• Log in to your Databricks workspace.
• In the sidebar, click Compute.
• In the list of available clusters, click the target cluster’s name.
• On the Apps tab, click Web Terminal. This should open up a terminal session tab in the web browser.
• Create a Databricks secrets scope using databricks secrets create-scope <scope-name>. For example, databricks secrets create-scope gg.
• Create a secret using

  databricks secrets put-secret --json '{
    "scope": "<scope-name>",
    "key": "<key-name>",
    "string_value": "<secret>"
    }'

  For example:

  databricks secrets put-secret --json '{
    "scope": "gg",
    "key": "storageaccountname",
    "string_value": "---------storage-account-key-----------------"
    }'

  Note:

  These commands were run using Databricks Runtime (DBR) 15.0.

  <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>databricks-jdbc</artifactId>
            <version>2.6.36</version>
        </dependency>

• If staging location is set to ADLS Gen2, classpath should include the ABS Event handler dependencies. See ABS Event handler dependencies.
• If staging location is set to S3, classpath should include the S3 Event handler dependencies. See S3 Event Handler.
• If staging location is set to GCS, classpath should include the GCS Event handler dependencies. See GCS Event Handler.

• Operation records can be aggregated in-memory, this is the default configuration.

• User can tune the frequency of merge interval using the File writer gg.handler.databricks.fileRollIntervalproperty, the default value is set to 3m (three minutes).

• Operation aggregation in-memory requires additional JVM memory configuration.

• To use SQL aggregation, it is mandatory that the trail files contain uncompressed UPDATE operation records, which means that the UPDATE operations contain full image of the row being updated.
• Operation aggregation using SQL can provide better throughput if the trails files contains uncompressed update records.
• Replicat can aggregate operations using SQL statements by setting the gg.aggregate.operations.using.sql=true.
• User can tune the frequency of merge interval using the File writer gg.handler.databricks.fileRollInterval property, the default value is set to 3m (three minutes).
• Operation aggregation using SQL does not require additional JVM memory configuration.

In some use cases, if the trail contains uncompressed update records, then the MERGE SQL statement can be optimized for better performance by setting gg.compressed.update=false

If you want to use DELETE+INSERT SQL statements instead of a MERGE SQL statement, then set gg.eventhandler.databricks.deleteInsert=true.

# Configuration to load GoldenGate trail operation records into Databricks using ADLS Gen2 stage.
# Note: Recommended to only edit the configuration marked as  TODO
gg.target=databricks
# Azure Blob Event handler.
gg.eventhandler.abs.bucketMappingTemplate=<azure_adls_gen2_container_name>
gg.eventhandler.abs.accountName=<azure_storage_account_name>
gg.eventhandler.abs.accountKey=<azure_storage_account_key>

# Databricks Event Handler.
#TODO: Edit JDBC ConnectionUrl
gg.eventhandler.databricks.connectionURL=jdbc:databricks://<server-hostname>:443;httpPath=<http-path>[;<setting1>=<value1>;<setting2>=<value2>;<settingN>=<valueN>];EnableArrow=0
#TODO: Edit JDBC username or 'token'
gg.eventhandler.databricks.UserName=token
#TODO: Edit JDBC password
gg.eventhandler.databricks.Password=<password>

#TODO: Edit the classpath to include Azure Blob Event Handler dependencies and Databricks JDBC driver.
gg.classpath=$THIRD_PARTY_DIR/abs/*:$THIRD_PARTY_DIR/databricks/*
#TODO: Provide sufficient memory (at least 8GB).
jvm.bootoptions=-Xmx8g -Xms1g

# Configuration to load GoldenGate trail operation records into Databricks using S3 stage.
# Note: Recommended to only edit the configuration marked as  TODO
gg.target=databricks
gg.stage=s3
#The S3 Event Handler
#TODO: Edit the AWS region
gg.eventhandler.s3.region=<aws region>
#TODO: Edit the AWS S3 bucket
gg.eventhandler.s3.bucketMappingTemplate=<s3 bucket>

# Databricks Event Handler.
#TODO: Edit JDBC ConnectionUrl
gg.eventhandler.databricks.connectionURL=jdbc:databricks://<server-hostname>:443;httpPath=<http-path>[;<setting1>=<value1>;<setting2>=<value2>;<settingN>=<valueN>];EnableArrow=0
#TODO: Edit JDBC username or 'token'
gg.eventhandler.databricks.UserName=token
#TODO: Edit JDBC password
gg.eventhandler.databricks.Password=<password>

#TODO: Edit the classpath to include GCS Event Handler dependencies and Databricks JDBC driver.
gg.classpath=$THIRD_PARTY_DIR/s3/*:$THIRD_PARTY_DIR/databricks/*
#TODO: Provide sufficient memory (at least 8GB).
jvm.bootoptions=-Xmx8g -Xms1g

# Configuration to load GoldenGate trail operation records into Databricks using GCS stage.
# Note: Recommended to only edit the configuration marked as  TODO
gg.target=databricks
gg.stage=gcs
## The GCS Event handler
#TODO: Edit the GCS bucket name
gg.eventhandler.gcs.bucketMappingTemplate=<gcs bucket>
#TODO: Edit the GCS credentialsFile
gg.eventhandler.gcs.credentialsFile=<oggbd-project-credentials.json>

# Databricks Event Handler.
#TODO: Edit JDBC ConnectionUrl
gg.eventhandler.databricks.connectionURL=jdbc:databricks://<server-hostname>:443;httpPath=<http-path>[;<setting1>=<value1>;<setting2>=<value2>;<settingN>=<valueN>];EnableArrow=0
#TODO: Edit JDBC username or 'token'
gg.eventhandler.databricks.UserName=token
#TODO: Edit JDBC password
gg.eventhandler.databricks.Password=<password>

#TODO: Edit the classpath to include GCS Event Handler dependencies and Databricks JDBC driver.
gg.classpath=$THIRD_PARTY_DIR/gcs/*:$THIRD_PARTY_DIR/databricks/*
#TODO: Provide sufficient memory (at least 8GB).
jvm.bootoptions=-Xmx8g -Xms1g