kafka.md

Kafka Architecture Overview

1. Kafka has high throughput but the storage time is very less. However, the storage time (retention period) can be configured based on the use case, and data can be stored for days, weeks, or indefinitely if there is enough storage.
2. Kafka has multiple producers and multiple consumers.
3. Inside Kafka, we have "topics".
4. Each topic is divided into partitions.
5. The user can set multiple rules to create different partitions.
6. Let's say we have a location topic, and the user can have two partitions based on the North Pole and South Pole.
7. Each consumer can subscribe to a partition.
8. Partitions work based on index, so if we have 3 partitions, the first one will be 0, the second one will be 1, and the third one will be 2.
9. Let's say we have one producer for a topic. Inside the topic, there are 4 partitions.
   • One Consumer: All the partition data will be transferred to the one and only consumer.
   • Two Consumers: Each consumer will receive data from 2 partitions.
   • Three Consumers: One consumer will receive data from 2 partitions, and the other two consumers will receive data from one partition each.
   • Four Consumers: Each consumer will receive data from one partition.
   • Five Consumers: Only 4 consumers will receive data from one partition each, and the 5th consumer will not receive any data.
10. The rule is: one consumer can track multiple partitions, but each partition can be tracked by only one consumer within a group.
11. Now, we can have consumer groups.
12. Any consumer won't exist without being in a group.
13. Let's say we have two groups, and each group has 2 consumers. If we have 4 partitions, each consumer from each group will receive data from 2 partitions. The rules mentioned in point 9 are unique for each group.
14. By having consumer groups, Kafka can act as a Queue or Pub/Sub system.
    • If we have 4 partitions and 1 consumer group with 4 consumers inside, Kafka acts like a Queue (each consumer gets a unique partition).
    • In the same scenario, if we create multiple consumer groups, it acts like a Pub/Sub (each group gets its own copy of the data).

Additional Points

15. Each partition has a replication factor for fault tolerance. If a broker fails, a replica becomes the new leader to keep the data available.
16. When consumers join or leave a group, Kafka rebalances the partitions among the active consumers. This ensures load is balanced but can cause temporary disruption.
17. Consumers track their position in partitions using offsets. These offsets can be stored by Kafka, allowing consumers to pick up from where they left off in case of failure.
18. Data inside a single partition is always ordered. Across partitions, data order is not guaranteed.
19. Consumer lag is the difference between the latest offset and the current consumer's offset. It's important to monitor lag to ensure real-time processing.
20. Producers can send messages with keys. The key determines which partition the message goes to, so all messages with the same key go to the same partition, which helps maintain order for specific data (e.g., all data for one user or location).
21. Kafka uses different partition assignment strategies like range and round-robin to distribute partitions among consumers in a group.
22. Adding more partitions can help scale the topic across more consumers, but more partitions add complexity and overhead. The number of partitions should match the expected number of consumers for optimal performance.