understanding about Kafka

kafka_handson.md

Apache kafka

Platform for real-time distributed streaming which just behaves like any other MESSAGING system

Design goals

• Scalability
• High Volume
• Data Transformations
• Low Latency
• Fault Tolerant

Factors of Kafka framework which forms a cluster

• Topic
• Producers
• Consumers
• Stream processors
• Connectors

---

Handson using kafka docker image

This will pull the fast-data-dev image from lensesio and will have the image runnable

$ docker run --rm --net=host lensesio/fast-data-dev 

List all the topics

$ kafka-topics --zookeeper 127.0.0.1 --list

List all the topics without zookeeper

$ kafka-topics --bootstrap-server 127.0.0.1:9092 --list

Create a topic in the kafka broker

$ kafka-topics --zookeeper 127.0.0.1 --topic pap --create

create topic with partitions and replication-factor

$ kafka-topics --zookeeper 127.0.0.1 -topic pap --create --partitions 3 --replication-factor 1

Describe the details about topic

$ kafka-topics --zookeeper 127.0.0.1 --topic pap --describe

Delete a topic

$ kafka-topics --zookeeper 127.0.0.1 --topic pap --delete

Producer which produce the data

$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic pap

Consumer to consume the data

$ kafka-console-consumer --bootstrap-server 127.0.0.1:9092  --topic pap --from-beginning

List all the consumer groups

$ kafka-consumer-groups  --bootstrap-server localhost:9092 --list

Consume the data from a specific topic and a specific group

$ kafka-consumer-groups --bootstrap-server localhost:9092 --group myPap --reset-offsets --to-earliest --execute --topic pap

Delete a kafka topic without using zookeeper reference

$ kafka-topics --bootstrap-server localhost:9092 --topic pap --delete

added kafka dependency