Connecting to Oracle Streaming Service

31 Connecting to Oracle Streaming Service

Oracle Streaming Service (OSS) supports putting messages to and receiving messages from OSS using the Kafka client. Therefore, Oracle GoldenGate for Big Data can be used to publish change data capture operation messages to OSS. You can use either the Kafka Handler or the Kafka Connect Handler.

The Kafka Connect Handler only supports using the JSON Kafka Connect converter. The Kafka Connect Avro converter is not supported because the Avro converter requires connectivity to a schema registry.

Note:

The Oracle Streaming Service currently does not have a schema registry to which the Kafka Connect Avro converter can connect. Streams to which the Kafka Handlers or the Kafka Connect Handlers publish messages must be pre-created in Oracle Cloud Infrastructure (OCI). Using the Kafka Handler to publish messages to a stream in OSS which does not already exist results in a runtime exception.

To create a stream in OCI, in the OCI console. select Analytics, click Streaming, and then click Create Stream. Streams are created by default in the DefaultPool.

Figure 31-1 Example Image of Stream Creation
The Kafka Producer client requires certain Kafka producer configuration properties to connect to OSS streams. To obtain this connectivity information, click the pool name in the OSS panel. If DefaultPool is used, then click DefaultPool in the OSS panel.

Figure 31-2 Example OSS Panel showing DefaultPool

Figure 31-3 Example DefaultPool Properties
The Kafka Producer also requires an AUTH-TOKEN (password) to connect to OSS. To obtain an AUTH-TOKEN go to the User Details page and generate an AUTH-TOKEN. AUTH-TOKENs are only viewable at creation and are not subsequently viewable. Ensure that you store the AUTH-TOKEN in a safe place.

Figure 31-4 Auth-Tokens

Once you have these configurations, you can publish messages to OSS.

For example, kafka.prm file:

replicat kafka
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.properties
map *.*, target qatarget.*;

Example:

kafka.properties
                file

gg.log=log4j 
gg.log.level=debug
gg.report.time=30sec
gg.handlerlist=kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.kafkaProducerConfigFile=oci_kafka.properties
# The following dictates how we'll map the workload to the target OSS streams
gg.handler.kafkahandler.topicMappingTemplate=OGGBD-191002
gg.handler.kafkahandler.keyMappingTemplate=${tableName}
gg.classpath=/home/opc/dependencyDownloader/dependencies/kafka_2.2.0/*
jvm.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar:dirprm

Example Kafka Producer Properties (oci_kafka.properties)

bootstrap.servers=cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="paasdevgg/oracleidentitycloudservice/user.name@oracle.com/ocid1.streampool.oc1.phx.amaaaaaa3p5c3vqa4hfyl7uv465pay4audmoajughhxlsgj7afc2an5u3xaq" password="YOUR-AUTH-TOKEN";

To view the messages, click Load Messages in OSS.

Figure 31-5 Viewing the Messages