9.2.29 OCI Streaming

Oracle Cloud Infrastructure Streaming (OCI Streaming) supports putting messages to and receiving messages using the Kafka client. Therefore, Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) can be used to publish change data capture operation messages to OCI Streaming.

You can use either the Kafka Handler or the Kafka Connect Handler. The Kafka Connect Handler only supports using the JSON Kafka Connect converter. The Kafka Connect Avro converter is not supported because the Avro converter requires connectivity to a schema registry.

Note:

The Oracle Streaming Service currently does not have a schema registry to which the Kafka Connect Avro converter can connect. Streams to which the Kafka Handlers or the Kafka Connect Handlers publish messages must be pre-created in Oracle Cloud Infrastructure (OCI). Using the Kafka Handler to publish messages to a stream in OSS which does not already exist results in a runtime exception.
  • To create a stream in OCI, in the OCI console. select Analytics, click Streaming, and then click Create Stream. Streams are created by default in the DefaultPool.

    Figure 9-2 Example Image of Stream Creation

    Streams are created by default in the “DefaultPool”
  • The Kafka Producer client requires certain Kafka producer configuration properties to connect to OSS streams. To obtain this connectivity information, click the pool name in the OSS panel. If DefaultPool is used, then click DefaultPool in the OSS panel.

    Figure 9-3 Example OSS Panel showing DefaultPool

    DefaultPool of the OSS Panel

    Figure 9-4 Example DefaultPool Properties

    Kafka Connection Settings for Stream Pool
  • The Kafka Producer also requires an AUTH-TOKEN (password) to connect to OSS. To obtain an AUTH-TOKEN go to the User Details page and generate an AUTH-TOKEN. AUTH-TOKENs are only viewable at creation and are not subsequently viewable. Ensure that you store the AUTH-TOKEN in a safe place.

    Figure 9-5 Auth-Tokens

    AUTH_TOKEN Creation

Once you have these configurations, you can publish messages to OSS.

For example, kafka.prm file: 

replicat kafka
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.properties
map *.*, target qatarget.*;
Example: kafka.properties file:
gg.log=log4j 
gg.log.level=debug
gg.report.time=30sec
gg.handlerlist=kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.kafkaProducerConfigFile=oci_kafka.properties
# The following dictates how we'll map the workload to the target OSS streams
gg.handler.kafkahandler.topicMappingTemplate=OGGBD-191002
gg.handler.kafkahandler.keyMappingTemplate=${tableName}
gg.classpath=/home/opc/dependencyDownloader/dependencies/kafka_2.2.0/*
jvm.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar:dirprm

Example Kafka Producer Properties (oci_kafka.properties) 

bootstrap.servers=cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="paasdevgg/oracleidentitycloudservice/user.name@oracle.com/ocid1.streampool.oc1.phx.amaaaaaa3p5c3vqa4hfyl7uv465pay4audmoajughhxlsgj7afc2an5u3xaq" password="YOUR-AUTH-TOKEN";

To view the messages, click Load Messages in OSS.

Figure 9-6 Viewing the Messages

Messages can be viewed in OSS by pressing the “Load Messages” command button.

Parent topic: Target