Kafka SSL Setup and Verification

A.1 Kafka SSL Setup and Verification

To configure Kafka SSL, follow below steps:

Verify that /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/config/server.properties, password, ssl truststore and keystorelocation is correct.

Figure A-1 Verify Server Properties

Description of "Figure A-1 Verify Server Properties"
Verify /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/config/ssl.properties, ssl.truststore.password, username, password is correct.

Figure A-2 Verify SSL Properties

Description of "Figure A-2 Verify SSL Properties"

Update kafka properties for product services. Execute below queries on PLATO schema (replace machine name and port as per env).

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:9092' where key like '%spring.cloud.stream.kafka.binder.brokers%' and application like 'obcr-%';

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181' where key like '%spring.cloud.stream.kafka.binder.zknodes%' and application like 'obcr-%';

In case, you are running Foundation and Product on two separate VMs, please copy the Kafka certificates present at the location /scratch/ssl/kafka_cert on the Foundation machine to /scratch/ssl/kafka_cert location on the Product machine. (Create the folder location if not already present and provide ownership of the kafka_cert folder to the VM user using below command).
```
cd /scratch/ssl
chown -R <VM user> kafka_cert
```
Login to the OBCR product weblogic and restart both the managed servers obcr_server1 and obcr_server2.

Troubleshooting

Verify the start log at location /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin/start_server.log (if SSL handshake error exists, then follow step 2)
Login to OBMA foundation weblogic, restart server Plato_Others_Server1 and CMC_Server4 which contains below services.
- plato-alerts-management-services
- plato-batch-servers
- cmc-advice-services
Verify the log at /scratch/work_area/logs to confirm that kafka error no more exists.

Restart Kafka

If for any reason Kafka restart is required, you need to stop zookeeper and kafka and then restart Kafka.

To restart Kafka:

In Putty, go to location /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin and run the commands as given below.
1. To stop Kafka run the below command twice:
```
./kafka-server-stop.sh
```
  The message Verify: No Kafka server to stop appears.
2. To stop Zookeeper run the below command twice:
```
./zookeeper-server-stop.sh
```
  The message Verify: No zookeeperserver to stop appears.
3. If still Kafka or Zookeeper does not stop, run the below command to stop Kafka:
  - ```
  ps -ef|grep kafka_2.13-3.4.0
```
  For specific process ID use the below command:
```
  jps | grep Kafka | awk '{print $1}'
```
  Once you run the command, kill the process ID with below command:
```
  kill -9 <process ID>
```
- ```
ps -ef|grep zookeeper
```
    For specific process ID use the below command:
```
ps aux | grep "org.apache.zookeeper.server.quorum.QuorumPeerMain" | grep -v grep | awk '{print $2}'
```
    Once you run the command, kill the process ID with below command:
```
kill -9 <process ID>
```
In Putty, go to this location /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin and start zookeeper using command.
```
nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &
```

In Putty, go to location /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin and and start kafka using command:

nohup /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin/kafka-server- start.sh /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/config/ server.properties > /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin/ start_server.log&

If the above steps don’t work and kafka is still not running, then perform the below steps:

Stop the zookeeper.
Delete kafka logs from this location (/scratch/obma/kafka_zookeeper/logs and /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/logs) and zookeeper logs (/tmp/zookeeper).
Start the zookeeper using the command mentioned in the above section.

In Putty, go to the location /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin and execute the below commands (replace machine name and the password in this command command)

./kafka-configs.sh --zookeeper ofss-mum- xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181 --alter --add- config "SCRAM-SHA-256=[password=admin-secret],SCRAM-
SHA-512=[password=admin-secret]" --entity-type users --entity-name admin

./kafka-configs.sh --zookeeper ofss-mum- xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181 --alter --add- config "SCRAM-SHA-256=[password=obedx-secret],SCRAM-
SHA-512=[password=obedx-secret]" --entity-type users --entity-name obcr

Start the kafka using the command mentioned in the above section.

Verify Kafka is Up

Execute below command:
```
ps -ef|grep kafka_2.13-3.4.0
```
It should show pid running.
Check logs at scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/bin/start_server.log. No SSL error should exist in this file.
Check logs at /scratch/obma/kafka_zookeeper/kafka_2.13-3.4.0/logs. It should display topic names.

Alternatively, you can check using below commands:

netstat -tlnp | grep :9092

Verify Kafka Health

Run the below command and verify:

$ netstat -tlnp | grep 9092

Note:

9092 is default port of kafka

Verify Zookeeper Health

Kafka instance will not start if Zookeeper is not yet started.

Run the below command and verify.

$ netstat tlnp | grep :2181 (2181 is default port of zookeeper) top6 0 0 :::2181 :::* LISTEN 19936/java

To debug, check if the permissions of Kafka log folder are correct.
The log folder path can be found by looking at the value of the property log-dirs in the server.properties file of Kafka installation.

Parent topic: Kafka Related Details