The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos, see https://cloud.oracle.com/en_US/cloud-infrastructure.

You can use any format handler that the File Writer Handler supports.

The Oracle Cloud Infrastructure Event Handler requires the Oracle Cloud Infrastructure Java software development kit (SDK) to transfer files to Oracle Cloud Infrastructure Object Storage. Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) does not include the Oracle Cloud Infrastructure Java SDK, see https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm.

You must download the Oracle Cloud Infrastructure Java SDK at:

https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/javasdk.htm

Extract the JAR files to a permanent directory. There are two directories required by the handler, the JAR library directory that has Oracle Cloud Infrastructure SDK JAR and a third-party JAR library. Both directories must be in the gg.classpath.

Specify the gg.classpath environment variable to include the JAR files of the Oracle Cloud Infrastructure Java SDK.

Example

gg.classpath=/usr/var/oci/lib/*:/usr/var/oci/third-party/lib/*

Setting of the proxy server settings requires additional dependency libraries identified by the following Maven coordinates:

Group ID: com.oracle.oci.sdk

Artifact ID: oci-java-sdk-addons-apache

The best way to get all of the dependencies is to use the Dependency Downloading utility scripts. The OCI script downloads both the OCI Java SDK and the Apache Addons libraries.

For more information on this dependency, see OCI Documentation - README.

You configure the Oracle Cloud Infrastructure Event Handler operation using the properties file. These properties are located in the Java Adapter properties file (and not in the Replicat properties file).

The Oracle Cloud Infrastructure Event Handler works only in conjunction with the File Writer Handler.

To enable the selection of the Oracle Cloud Infrastructure Event Handler, you must first configure the handler type by specifying gg.eventhandler.name.type=oci and the other Oracle Cloud Infrastructure properties as follows:

Sample Configuration

gg.eventhandler.oci.type=oci
gg.eventhandler.oci.configFilePath=~/.oci/config
gg.eventhandler.oci.profile=DEFAULT
gg.eventhandler.oci.namespace=dwcsdemo
gg.eventhandler.oci.region=us-ashburn-1
gg.eventhandler.oci.compartmentID=ocid1.compartment.oc1..aaaaaaaajdg6iblwgqlyqpegf6kwdais2gyx3guspboa7fsi72tfihz2wrba
gg.eventhandler.oci.pathMappingTemplate=${schemaName}
gg.eventhandler.oci.bucketMappingTemplate=${schemaName}
gg.eventhandler.oci.fileNameMappingTemplate=${tableName}_${currentTimestamp}.txt
gg.eventhandler.oci.finalizeAction=NONE
goldengate.userexit.writers=javawriter

Basic configuration information like user credentials and tenancy Oracle Cloud IDs (OCIDs) of Oracle Cloud Infrastructure is required for the Java SDKs to work, see https://docs.cloud.oracle.com/iaas/Content/General/Concepts/identifiers.htm.

The ideal configuration file include keys user, fingerprint, key_file, tenancy, and region with their respective values. The default configuration file name and location is ~/.oci/config.

Create the config file as follows:

1. Create a directory called .oci in the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) home directory
2. Create a text file and name it config.
3. Obtain the values for these properties:

   user

   1. Login to the Oracle Cloud Infrastructure Console https://console.us-ashburn-1.oraclecloud.com.
   2. Click Username.
   3. Click User Settings.

      The User's OCID is displayed and is the value for the key user.

   tenancy

   The Tenancy ID is displayed at the bottom of the Console page.

   region

   The region is displayed with the header session drop-down menu in the Console.

   fingerprint

   To generate the fingerprint, use the How to Get the Key's Fingerprint instructions at:

   https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

   key_file

   You need to share the public and private key to establish a connection with Oracle Cloud Infrastructure. To generate the keys, use the How to Generate an API Signing Keyat:

   https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

   pass_phrase

   This is an optional property. It is used to configure the passphrase if the private key in the pem file is protected with a passphase. The following openssl command can be used to take an unprotected private key pem file and add a passphrase.

   The following command prompts the user for the passphrase:

   openssl rsa -aes256 -in in.pem -out out.pem

Sample Configuration File

user=ocid1.user.oc1..aaaaaaaat5nvwcna5j6aqzqedqw3rynjq
fingerprint=20:3b:97:13::4e:c5:3a:34
key_file=~/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaaba3pv6wkcr44h25vqstifs

Connectivity Issues

If the OCI Event Handler is unable to connect to the OCI object storage when running on premise, it’s likely your connectivity to the public internet is protected by a proxy server. Proxy servers act a gateway between the private network of a company and the public internet. Contact your network administrator to get the URL of your proxy server.

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) connectivity to OCI can be routed through a proxy server by setting the following configuration properties:

gg.eventhandler.name.proxyServer={insert your proxy server name}
gg.eventhandler.name.proxyPort={insert your proxy server port number}

ClassNotFoundException Error

The most common initial error is an incorrect classpath that does not include all the required client libraries so results in a ClassNotFoundException error. Specify the gg.classpath variable to include all of the required JAR files for the Oracle Cloud Infrastructure Java SDK, see Detailing the Functionality.

Duplicate records after Replicat Recovery

OCI Object Storage handler replication uses File Writer Handler and OCI Object Storage handler in the replicat. Oracle GoldenGate prioritizes no data loss and guarantees no data loss in case of failures by at least once semantics in OCI Object Storage (json, csv, delimtedtext, avro_orc, parquet) delivery. In the cases if replicat runs fine and normally shut down, then exactly once is supported. In case of failures (because of network failures), there are various reason that can lead into duplicates in recovery.

Two cases where duplicates can occur:

1. If data is written and a failure occurs between when the data is written, and when the checkpoint is moved. Then upon restart the replicat backs up to the previous checkpoint and data can unfortunately be replayed.
2. The rolling of the data files occurs based on customer configured triggers. Trigger can be file size, time, inactivity, or time of day. The rolling does not necessarily happen on a transaction commit boundary. The trigger causes writing to the current file to complete, the post processing transformation and movement complete, and any state on that file is deleted. If a replicat abend occurs in between when the rolling is processed and when the checkpoint is moved, then upon restart, it can again replay those messages.

If you observe duplicate records in case of OCI Object Storage replicat recovery, then it is an expected behavior. If you observe duplicates while replicat is running fine, then file a support ticket.