getting started with MSK Kafka

getting started with MSK Kafka.md

Getting Started with AWS MSK Kafka

Create Kafka MSK

Create AWS kafka instance you can use default broker sige for this POC.

Create an EC2 Instance

Ec2 used for publisher and subscriber of kafka events.

Setup your EC2 Instance to produce Kafka Topic

1. Kafka needs Java your EC2 will not have Java installed. Therefore, please install Java. Run below script on EC2 to install java. Make sure the EC2 instance security group is added in MSK and vice versa.

sudo yum install java-1.8.0

2. Download apache kafka on EC2

wget https://archive.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz

3. Extract kafka directory on EC2

tar -xzf kafka_2.12-2.2.1.tgz

Create Topic on MSK from EC2 instance

Lets create topic called AWSKafkaTutorialTopic. In order to create topic we will use Apache zookeeper in kafka. Therefore, go to Kafka client information and get the apache zookeeper connection in plaintext. Make sure in your EC2 AWS Cli is installed.

In order to create message you need to make sure the EC2 security group Id is added as a rule in the kafka msk cluster inbound rules. Therefore, go to MSK security group click on edit inbound rules and add new rule allow all traffic from security group id of your EC2 instance.

• Connect to EC2 instance
• Go to cd kafka_2.12-2.2.1/
• Edit & Execute below script to create topic on Kafka using zookeeper

bin/kafka-topics.sh --create --zookeeper "z-3.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:2181,z-2.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:2181,z-1.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:2181" --replication-factor 2 --partitions 1 --topic AWSKafkaTutorialTopic

Next we will produce and consume messages.

Setup your EC2 instance to produce Messages

Please note step1 and Step2 you do not need if you want to send messages in plain text.

Step1: Use the Trust store to create messages in EC2 instance

Note: If you do not want to use TLS then this step is not required.

Copy over some of our Java Folders in the trust store of the Kafka in the EC2 instance to produce messages.

# Connect EC2 instance go to jvm folder and copy the java runtime name
cd /usr/lib/jvm

# in my case this is the name so use this in below script java-1.8.0-openjdk-1.8.0.312.b07-1.amzn2.0.2.x86_64 
cp /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.amzn2.0.2.x86_64/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks 

Step2: Update kafka client properties in EC2 Instance

Note: If you do not want to use TLS then this step is not required. We have to create a client properties file that contains security protocols on the trust store location that is the temp directory location that we just created /tmp/kafka.client.truststore.jks and copied jar files. It will create a folder details kafka which will tell where the client trust stores is from JAVA.

• Connect to EC2 Instance
• Go to the bin folder of the cd kafka_2.12-2.2.1/bin
• Open client.properties with Vim by running script vim client.properties the type "i" to go in insert mode. Then enter below 2 lines.

security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks

Next hit esc Next type :x to save and exit.

Step3: Get Kafka Bootstrap Brokers TLS String

b-1.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-2.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-3.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094

Step4: Execute Producer Script in EC2 to create messages

In order to produce messages from EC2 instance you need kafka bootsrapping broker TLS string. Note you can use plain text string if you are not doing encryption.

TLS String: b-1.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-2.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-3.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094

Let's create some messages for topic AWSKafkaTutorialTopic Connect your EC2 instance Go to the bin folder of the kafka_2.12-2.2.1

cd kafka_2.12-2.2.1/bin

Script Template to produce Message

./kafka-console-producer.sh --broker-list BootstrapBrokerStringTls --producer.config client.properties --topic AWSKafkaTutorialTopic

Run below example Script to create message, enter your bootstrap broker TLS string

./kafka-console-producer.sh --broker-list b-1.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-2.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-3.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094 --producer.config client.properties --topic AWSKafkaTutorialTopic

Next lets enter some text that will become message.

Comsuming Kafka Messages in EC2 instance

In my case, I am using same EC2 instance to subscribe to the message. Therefore, connect to the EC2 instance.

Create subscriber for the message and receive them.

Open new terminal & Go to the bin folder of the kafka_2.12-2.2.1

Template script to consume Message

./kafka-console-consumer.sh --bootstrap-server BootstrapBrokerStringTls --consumer.config client.properties --topic AWSKafkaTutorialTopic --from-beginning

Run below example script to receive messages (enter your bootstrap broker TLS string)

./kafka-console-consumer.sh --bootstrap-server b-1.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-2.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094,b-3.sdp-bbg-poc.smw3wi.c20.kafka.us-east-1.amazonaws.com:9094 --consumer.config client.properties --topic AWSKafkaTutorialTopic --from-beginning

Reference

https://www.youtube.com/watch?v=y4wowEQd4Os