Kafka Setup with UI dashboard

docker-compose-kafka.yml

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - '2181:2181'

  kafka:
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE  # Set this to INSIDE or OUTSIDE
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - '9092:9092'
      - '9094:9094'
    depends_on:
      - zookeeper

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
      AUTH_TYPE: "LOGIN_FORM"
      SPRING_SECURITY_USER_NAME: admin
      SPRING_SECURITY_USER_PASSWORD: pass
    depends_on:
      - kafka

Kafka with Zookeeper and UI Setup

This guide helps you run Kafka, Zookeeper, and Kafka-UI locally using Docker Compose and provides instructions on how to use the Kafka-UI and connect Kafka from external services.

---

Prerequisites

• Install Docker
• Install Docker Compose

---

Steps to Run Kafka and UI

1. Save the docker-compose.yml File:

   Create a file named docker-compose.yml and paste the gist content

2. Start the Services:

   Run the following command in the same directory as your docker-compose.yml file:

   docker-compose up -d

3. Verify the Services:

   • Zookeeper: Ensure it is running on port 2181.
   • Kafka: Check ports 9092 (internal) and 9094 (external).
   • Kafka-UI: Open http://localhost:8080 in your browser. Login with:
     • Username: admin
     • Password: pass

---

Using Kafka-UI

1. Login to the UI:
   Navigate to http://localhost:8080 and enter the credentials.

2. Browse Topics:

   • View, create, and delete Kafka topics.
   • View message details within a topic.

3. Send Messages:

   • Use the UI to produce messages to a topic.

4. Monitor Consumers:

   • Inspect active consumer groups and their offsets.

---

Connecting Kafka from External Services

To connect an external service to Kafka, use the advertised listener OUTSIDE://localhost:9094.

Here’s an example of connecting from a Node.js application using the kafkajs library:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9094'],
});

const producer = kafka.producer();

async function sendMessage() {
  await producer.connect();
  await producer.send({
    topic: 'my-topic',
    messages: [
      { value: 'Hello Kafka!' },
    ],
  });
  await producer.disconnect();
}

sendMessage().catch(console.error);

---

Stopping the Services

To stop the containers, run:

docker-compose down

---

Notes

• Adjust environment variables if you want to customize the setup (e.g., credentials, ports).
• Ensure port 9094 is accessible from external services.
• If running on a remote machine, replace localhost with the machine's IP address in KAFKA_ADVERTISED_LISTENERS and your service configurations.

Kafka setup in KRAFT Mode

This setup doesn't require zookeeper (Not recommended for production)

version: '3.8'

services:
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-kraft
    environment:
      CLUSTER_ID: "kraft-cluster-1"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "controller,broker"
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094,CONTROLLER://0.0.0.0:9093
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@localhost:9093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    ports:
      - '9092:9092'
      - '9094:9094'
      - '9093:9093' # Add port for controller listener

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
      AUTH_TYPE: "LOGIN_FORM"
      SPRING_SECURITY_USER_NAME: admin
      SPRING_SECURITY_USER_PASSWORD: pass
    depends_on:
      - kafka

How to use it

Everything will be the same as the above with zookeeper setup for deploying and way to connect to the broker