11.2.2 Kafka Producer/Consumer Configurations

OBDX provides some default configurations for Kafka for Broker settings, partitioning and replication, consumer group size, etc. maintained in the table DIGX_FW_CONFIG_ALL_B with category_id KAFKA_CONFIG. The configurations by default will be applicable to all producers and consumers.

1. Generic Configurations

Below mentioned are the generic configurations (PROP_ID) and their default values (PROP_VALUE). These can be overridden if required, by updating them in the table.

Table 11-3 (PROP_ID) and their default values

PROP_ID Default PROP_VALUE Description
bootstrap.servers localhost:8080 Specifies the Kafka broker(s) that consumers and producers should connect to.
enable.auto.commit true Determines whether the consumer's offset is automatically committed.
auto.commit.interval.ms 5000 The frequency (in milliseconds) at which the consumer commits offsets when auto-commit is enabled.
auto.offset.reset latest

Defines the behavior when a consumer starts reading from a topic. Options:

earliest: Read from the beginning of the log.

latest: Read only new messages.
CONSUMER_POLL_TIMEOUT_MS 2000 The maximum time (in milliseconds) a consumer waits for records when polling from Kafka.

Apart from the above mentioned properties, any other producer and consumer configuration provided by Kafka can also be overridden by adding the respective entry in the table DIGX_FW_CONFIG_ALL_B.

2. Changing Topic Level Configurations

Bank can also override any producer and consumer configuration for a particular topic with their custom values instead of the default ones, by adding an entry in the table DIGX_FW_CONFIG_ALL_B in the column PROP_ID with the pattern TOPIC_NAME@CONFIGURATION.

For example, 

Insert into DIGX_FW_CONFIG_ALL_B
          (PROP_ID,CATEGORY_ID,PROP_VALUE,FACTORY_SHIPPED_FLAG,PROP_COMMENTS,SUMMARY_TEXT,
CREATED_BY,CREATION_DATE,LAST_UPDATED_BY,LAST_UPDATED_DATE,OBJECT_STATUS,OBJECT_VERSION_NUMBER,
EDITABLE,CATEGORY_DESCRIPTION)
          values('structure-createdAndAuthorized@auto.offset.reset','KAFKA_CONFIG','latest','N',null,'Auto
          offset reset','ofssuser'sysdate,'ofssuser',sysdate,'A',1,'Y', 'Auto offset reset
          ');

Parent topic: Kafka