Utilizing the Local Kafka Cluster

3.1.2 Utilizing the Local Kafka Cluster

You can use the local Kafka cluster as a source or target in GGSA pipelines. The instance provides a range of Kafka utilities to read and write data from local Kafka. These utilities are in the README.txt file in the/u01/app/osa/utilities/kafka-utils folder. Below is the description of the scripts for quick reference:

For Kafka-related help, type ./kafka.sh help.
To list Kafka topics, run ./kafka-topics.sh.
To listen to a Kafka topic, type ./kafka-listen.sh {topic-name}.
To feed a csv file into Kafka, run

cat {your-csv-file} | ./csv2json.sh | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}
To feed a json file, run cat {your-json-file} | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}
To loop through your csv file, run ./loop-csv.sh {your-csv-file} | ./csv2json.sh | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}
The command sampler.sh {n} {t} reads {n} lines in each {t} seconds from STDIN and writes to STDOUT. Default values: n=1 t=1
The command csv2json.sh reads the CSV from the STDIN and writes JSON to STDOUT. First line must be the CSV header.
There are data files to simulate GoldenGate capture:

cat ./gg_orders.json | ./sampler.sh | ./kafka.sh feed goldengate
There are data files to simulate nested json structures:

./loop-file.sh ./complex.json | ./sampler.sh 1 1 | ./kafka.sh feed complex
There are data files to simulate movement of buses:

cat ./30-min-at-50-rps.json | ./sampler.sh 20 1 | kafka.sh feed buses
In general to feed test data to your running kafka installation, run {your-shell-script} | kafka.sh feed {your-topic-name}

Note:

your-shell-script generates data in JSON format.

Sample output:

{"price":284.03,"symbol":"USDHUF=x","name":"USD/HUF"}

{"price":316.51,"symbol":"EURHUF=x","name":"EUR/HUF"}

{"price":0.8971,"symbol":"USDEUR=x","name":"USD/EUR"}