Connect to Apache Iceberg

Learn to create a connection to Apache Iceberg in OCI GoldenGate.

Before you begin

Ensure that you:

Review how OCI GoldenGate connects to your source and targets.
Configure the required policies to enable secure Vault and Secrets access, such as use secrets, use vaults, and read secret-bundles. For more information, see Minimum recommended policies.

Create the connection

To create a connection to Apache Iceberg:

From the OCI GoldenGate Overview page, select Connections.

You can also select Create Connection under the Get started section and skip to step 3.
On the Connections page, select Create Connection.
On the Create Connection page, complete the fields as follows:
1. For Name, enter a name for the connection.
2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
3. (For GoldenGate on Multicloud only) Select your Subscription, and then complete the following fields.
  1. From the Compartment dropdown, select the compartment in which the Resource Anchor resides.
  2. Select the Multicloud partner region.
  3. Select your Partner availability zone. The available options populate based on the selected Multicloud partner region.
4. For Compartment, select the compartment in which to create the connection.
5. For Type, under Big Data, select Apache Iceberg.
6. Select a Catalog type from the dropdown, and then complete the relevant catalog fields:
  1. For Glue, enter the Glue ID.
  2. For Hadoop, no additional information is needed.
  3. For Nessie, specify:
    - URI: Enter the Nessie catalog URI.
    - Branch: Enter the active branch name from which Nessie reads and write table metadata.
  4. For Polaris, specify:
    - URI: Enter the Polaris catalog URI.
    - Name: Enter the name in which Polaris registers Iceberg tables.
    - Client ID: Enter the OAuth client ID to use for authentication.
    - Client secret: Select the client secret, or select Create client secret to create a new one. If you choose to create a new client secret, provide the following information:
      - Name
      - (Optional) Description
      - Select the Compartment in which the properties secret resides.
      - Select the Vaultin which to store the properties secret.
      - Select the Encryption key to use.
      - Enter the User Password, then confirm the User Password.
    - Principal role: Enter the Snowflake role used to access Polaris.
  5. For Rest, specify:
    - URI: Enter the base URL for the REST catalog API.
    - Properties Secret: Select the properties secret, or select Create properties secret to create a new one. If you choose to create a new properties secret, provide the following information:
      - Name
      - (Optional) Description
      - Select the Compartment in which the properties secret resides.
      - Select the Vault in which to store the properties secret.
      - Select the Encryption key to use.
      - Upload the Properties secret ID.
7. Select a Storage type from the dropdown, and then complete the relevant storage fields:
  - Oracle Object Storage (S3 API):
    1. Enter the Access key ID
    2. Select the Object Storage bucket.
    3. Create or select the Secret access key secret ID.
    4. Enter the Endpoint.
  - Amazon S3:
    1. Select a Scheme type.
    2. Enter the Amazon Region where the S3 bucket is located.
    3. Enter the S3 Bucket where Iceberg stores metadata and data files.
    4. Enter the Access key ID to access the S3 bucket.
  - Google Cloud Storage:
    1. Enter the Google Cloud Storage Bucket where Iceberg stores metadata and data files.
    2. Enter the Project ID where the bucket exists.
    3. For Endpoint, enter the url to connect to Google Cloud Storage.
  - Azure Data Lake Storage:
    1. Enter the Azure Storage Account name.
    2. Enter the Azure blob storage Container where Iceberg stores its tables.
  - None (REST Catalog type only)
8. Expand Advanced options. You can configure the following options:
  - Security
    - Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:
      - Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
      - Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
  - Network connectivity
    - Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
    - Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.
      Note:
      
      If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
      
      Learn more about Oracle GoldenGate connectivity.
  - Security attributes: Add security attributes to control access to this connection using Zero Trust Packet Routing (ZPR).
  - Tags: Add tags to organize your resources.
Select Create.

After the connection is created, it appears in the Connection list. Ensure that you assign the connection to a deployment to use it in a data replication process.

Next steps

Known issues

Apache Iceberg connection to AI Data Catalog service authentication error

Apache Iceberg connections to AI Data Catalog service expires due to authentication error caused by token expiration. If you encounter this issue, contact Oracle Support.

Apache Iceberg NoSuchFileException error

If you encounter the following error when using Apache Iceberg connections, open a support ticket, share the details and error message:

Caused by: java.nio.file.NoSuchFileException: /tmp/ogg/ocid1.goldengateconnection.oc1...../restCatalogProperties