Kafka – Part 6 – Configure Kafka Server instance in Pega

In another blog article on queue processor, we saw that Pega internally has its native Kafka server to perform stream processing. It is recommended to visit the Kafka articles in order for better understanding.

https://myknowtech.com/tag/kafka

In this blog article, we will see how Pega can make a connection to an external Kafka server.

This tutorial is implemented using Pega personal edition 8.4, but the core concepts remain the same in higher versions as well

Business scenario: A banking organization ABC uses Pega infinity to facilitate their sales process. Whenever a loan is issued, loan details are captured in a Java application that maintains the loan life cycle. Pega application is used to service customers who fails to pay the loan amount. The service case stores the loan details.

Business problem

Whenever a loan is fully settled, the loan status is updated in the golden source system – SOR (Java application) but the changes need to be propagated to all systems that use the loan status to make tem up-to-date.

Solution

Kafka is chosen as the messaging platform. Whenever the loan expires, the Java application produces loan status message in a dedicated topic. Pega application is registered as one of the consumer and consumes the loan status change messages.

As a first step, we need to make a connection between Pega and external Kafka server.

Mandatory pre-requisite – Please follow the below article, to set your external Kafka server in your local machine.

1. Start the Kafka external server

Step 1: As per the documentation, start the zoopkeeper using the below command.

You should see the zookeeper binding message.

Important note: Make sure your external kafka server binding port – 2181 do not collide with Pega zookeeper binding port. I am using 2181 for external Kafka and 2182 for Pega zookeeper service

Step 2: Start the Kafka server.

You should see the Kafka server started successfully.

Note: The external Kafka server runs on port 9093 and Pega Kafka server runs on 9092.

2. Make the connection from Pega to Kafka

Step 1: Login to Pega designer studio

Step 2: Create a new Kafka Instance

Records -> SysAdmin -> Kafka -> Create new

Step 3: Configure the Kafka instance details

Details –

In the details block you need to enter the host and port ( you can also add multiple hosts in a cluster) of the external Kafka server to connect to

I am using the localhost IP address – 127.0.0.1 and port 9093

Security settings –

In the security settings block, you have the option to enable either one-way or two-way SSL settings using the truststore and keystore instance.

Authentication –

When your Kafka cluster is configured to use the JAAS configuration file for authentication then you will prompted with fields to enter the credentials.

For more details, please visit the link

https://docs.confluent.io/current/kafka/authentication_sasl/index.html

Currently, for local machines, I didn’t implement any sort of security settings, but in real-time mostly you will be using it.

Step 4: Click test connectivity.

Connection successfully established between Pega and external Kafka.

As a summary,

– External Kafka is started on port 9093 with zookeeper on 2181.

– Kafka Instance is created in Pega and made a connection with External Kafka server.

In the next article, we will see how Pega can consume the published Kafka messages.