Creating a Kafka Connect Configuration

Create a Kafka Connect configuration in the Streaming service.

To review requirements for working with Kafka Connect see Using Kafka Connect.

    1. On the Kafka Connect Configurations list page, select Create Kafka Connect Configurations. If you need help finding the list page, see Listing Kafka Connect Configurations.
      The Create Kafka Connect Configuration dialog box opens.
    2. Select a compartment from the Compartment list.
    3. In the Kafka Connect Configuration Name box, enter a name for the configuration. Avoid entering confidential information.
    4. (Optional) Add one or more tags to the Kafka Connect configuration: Select Show Advanced Options to show the Add Tags section.
      If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
    5. Select Create Kafka Connect Configuration.
      The details page for the new Kafka Connect configuration opens. A read-only text box labeled Kafka Connect Storage Topics lists the connector configuration.
    6. Select Copy to copy the connector configuration so that you can paste it into the connect-distributed.properties file for your Kafka connector.
      For more information, see the official Kafka Connect documentation.
  • Use the oci streaming admin connect-harness create command and required parameters to create a Kafka Connect configuration:

    oci streaming admin connect-harness create --name <kafka_connect_configuration_name> --compartment-id <compartment_OCID>

    For example:

    oci streaming admin connect-harness create --name MyKafkaConnectConfig --compartment-id exampleuniqueID
    {
      "data": {
        "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
        "defined-tags": {},
        "freeform-tags": {},
        "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
        "lifecycle-state": "CREATING",
        "lifecycle-state-details": null,
        "name": "MyKafkaConnectConfig",
        "time-created": "2020-11-04T17:36:44.323000+00:00"
      },
      "etag": "\"48886b1f-333f-4142-a335-ed42e88b37f8-0e7d329d-c6df-47fc-b814-f1237b414281\""
    }

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

  • Run the CreateConnectHarness operation to create a Kafka Connect configuration.

Using the SDK for Java

The following code example shows how to create a Kafka Connect harness using the OCI SDK for Java:

CreateConnectHarnessDetails createConnectHarnessDetails = CreateConnectHarnessDetails.builder()
    .compartmentId(compartment) //compartment where you want to create connect harness
    .name("myConnectHarness") //connect harness name
    .build();
 
CreateConnectHarnessRequest connectHarnessRequest = CreateConnectHarnessRequest.builder()
    .createConnectHarnessDetails(createConnectHarnessDetails)
    .build();
 
CreateConnectHarnessResponse createConnectHarnessResponse = streamAdminClient.createConnectHarness(connectHarnessRequest);
ConnectHarness connectHarness = createConnectHarnessResponse.getConnectHarness();
 
while (connectHarness.getLifecycleState() != ConnectHarness.LifecycleState.Active && connectHarness.getLifecycleState() != ConnectHarness.LifecycleState.Failed) {
    GetConnectHarnessRequest getConnectHarnessRequest = GetConnectHarnessRequest.builder().connectHarnessId(connectHarness.getId()).build();
     connectHarness = streamAdminClient.getConnectHarness(getConnectHarnessRequest).getConnectHarness();
}

To use Kafka Connect with Streaming, you need a Kafka Connect configuration, or Kafka Connect harness. You can retrieve the OCID for a harness when you create a new harness or use an existing one. For more information, see Using Kafka Connect.