Topic & Partition Page

Select the operation and topic on which to perform the operation and optionally specify the message structure.

Configure the Confluent Adapter as an Invoke Connection

Element Description

Select a Topic

 Select the topic on which to perform the operation. You can also enter the beginning letters of the topic to filter the display of topics. A topic is a category in which applications can add, process, and reprocess messages. You subscribe to messages in topics.

Specify the Partition

(This field is only displayed if you select Publish records to a Kafka topic or Consume records from a Kafka topic.)

Specify the partition to which to push the selected topic. Kafka topics are divided into partitions that enable you to split data across multiple brokers. If you do not select a specific partition and use the Default selection, Kafka considers all available partitions and decides which one to use.
Consumer Group

(This field is only displayed if you select Consume records from a Kafka topic.)

Specify the consumer group to attach.

Consumers join a group by using the same group ID. Confluent assigns the partitions of a topic to the consumers in a group.

Note: When using a nondefault partition, specify the consumer group specific to the topic partition.

Specify the option for consuming messages

(This field is only displayed if you select Consume records from a Kafka topic.)

  • Read latest: Reads the latest messages starting at the time at which the integration was activated.
  • Read from beginning: Select to read messages from the beginning.

    As an example, if you select to read from the beginning and have activated the integration, the first scheduled run picks up 20 records and the next scheduled run picks up the next 20 records. If the integration is then deactivated, edited, and reactivated, the next 20 records are picked up.

  • Read from specific offset: Select to reset the consumer to a specific offset (partition-specific). Click Add to add the partition and the offset.
Once configured, if you edit it again, the following options are displayed:
  • Continue reading: Reads the next message from where it left off before deactivation.
  • Reset to beginning: Reads the message again from the beginning (may cause duplication).
  • Read from specific offset: Select to reset the consumer to a specific offset (partition-specific). Click Add to add the partition and the offset (may cause duplication).
Notes:
  • When you select Read from beginning, Read latest, or Read from specific offset, not all messages are guaranteed to be consumed in a single run. However, any remaining messages are consumed in subsequent runs.
  • Selecting Read from beginning, Reset to beginning, or Read from specific offset resets the consumer group offset upon completing the wizard.
Maximum Number of Records to be fetched

(This field is only displayed if you select Consume records from a Kafka topic or Consume records from a Kafka topic by specifying offset.)

Specify the number of messages to read. The threshold for the complete message payload is 10 MB.

Note: This field specifies the upper boundary of records to fetch. It does not guarantee the specified amount of records to retrieve from the stream in a single run. Remaining messages are fetched in subsequent runs.

Do you want to specify the message structure?

 Select Yes if you want to define the message structure to use on the Message Structure page of the wizard. Otherwise, select No.

Do you want to specify the headers for the message?

 Select Yes if you want to define the message headers to use on the Headers page of the wizard. Otherwise, select No.
Review and update advanced configurations Click Edit to open the Advanced Options section to enable or disable the transactional producer.
  1. Transaction Producer: This field is only displayed if you select Publish records to a Kafka topic. This option provides the following capabilities:
    1. If selected, the transactional producer enables an application to send messages to multiple partitions atomically.
    2. If not selected, the Apache Kafka Adapter is configured as a nontransactional producer.
  2. Message Type: This option defines the message type. Available options are String or Bytes. It defines the serializers to use for the message. This selection is applicable for the message key and value.

Configure the Confluent Adapter as a Trigger Connection

Element Description
Select a Topic Select the topic on which to perform the operation. You can also enter the beginning letters of the topic to filter the display of topics. A topic is a category in which applications can add, process, and reprocess messages. You subscribe to messages in topics.
Specify the Partition

(This field is only displayed if you are reading data from topics using the Kafka Consumer API.)

Specify the partition to which to push the selected topic. Kafka topics are divided into partitions that enable you to split data across multiple brokers. If you do not select a specific partition and use the Default selection, Kafka considers all available partitions and decides which one to use.
Consumer Group

(This field is only displayed if you are reading data from topics using the Kafka Consumer API.)

Specify the consumer group to attach.

Consumers join a group by using the same group ID. Confluent assigns the partitions of a topic to the consumers in a group.

Note: When using a nondefault partition, specify the consumer group specific to the topic partition.

Specify the option for consuming messages
  • Read latest: Reads the latest messages starting at the time at which the integration was activated.
  • Read from beginning: Select to read messages from the beginning.
  • Read from specific offset: Select to reset the consumer to a specific offset (partition-specific). Click Add to add the partition and the offset.
Once configured, if you edit it again, the following options are displayed:
  • Continue reading: Reads the next message from where it left off before deactivation.
  • Reset to beginning: Reads the message again from the beginning (may cause duplication).
  • Read from specific offset: Select to reset the consumer to a specific offset (partition-specific). Click Add to add the partition and the offset (may cause duplication).
Note:
  • When you select Read from beginning, Read latest, or Read from specific offset, not all messages are guaranteed to be consumed in a single run. However, any remaining messages are consumed in subsequent runs.
  • Selecting Read from beginning, Reset to beginning, or Read from specific offset resets the consumer group offset upon completing the wizard.
Application ID

This field is only displayed if you are reading data from topics using the Kafka Streaming API.)

 Specify the application ID to attach. The stream must have a unique ID.
Polling Frequency (Sec)

(This field is only displayed if you are reading data from topics using the Kafka Consumer API.)

Specify the frequency at which to fetch records.

Note: Trigger-based polling supports a maximum frequency interval of 10 minutes.

Maximum Number of Records to be fetched

(This field is only displayed if you are reading data from topics using the Kafka Consumer API)

Specify the number of messages to read. The threshold for the complete message payload is 10 MB.

Note: This field specifies the upper boundary of records to fetch. It does not guarantee the specified amount of records to retrieve from the stream in a single run. Remaining messages are fetched in subsequent runs.

Do you want to specify the message structure?

 Select Yes if you want to define the message structure to use on the Message Structure page of the wizard. Otherwise, select No.

Do you want to specify the headers for the message?

 Select Yes if you want to define the message headers to use on the Headers page of the wizard. Otherwise, select No.
Review and update advanced configurations Click Edit to open the Advanced Options section.
  1. Message Type: This option defines the message type. Available options are String or Bytes. It defines the serializers to use for the message. This selection is applicable for the message key and value.
Review and update rejected message configurations Define the topic for rejected messages. You can select the topic to which the rejected message is copied. You can also specify the partition under the selected topic.