How Apache Kafka Works
IntermediateApache Kafka is a distributed event streaming platform. Unlike traditional message queues that delete messages after delivery, Kafka stores messages durably as an ordered, immutable log — and any number of consumers can read the same log independently at their own pace. This design makes Kafka ideal for event-driven architectures, real-time data pipelines, and microservice communication at scale. Kafka can sustain millions of messages per second across a cluster while maintaining durability guarantees.
Think of Kafka as a commit log, not a post office
A traditional message queue is like a post office: it holds a letter until one recipient picks it up, then destroys it. Kafka is like a commit log in a database: every transaction is appended to an ordered, persistent log. Any process can read the log from any position — the 3rd entry, the 1000th, or the latest. Reading does not consume the entry. You can replay events from last week. Multiple teams can independently process the same stream. The log grows continuously; old entries are deleted only after a configurable retention period (e.g., 7 days).
Step by Step
Key Concepts
Topic & Partition
A topic is a named log category. Each topic is split into partitions — independent, ordered, append-only logs. Partition count determines maximum consumer parallelism: a topic with 12 partitions can be consumed by up to 12 consumers in a group simultaneously. Choose partition count thoughtfully — you can increase it later but this changes key-to-partition mapping and rebalances consumers.
Offset
A unique sequential ID for each message within a partition. Message at offset 5 is the 6th message in that partition (0-indexed). Consumers commit offsets to track progress. You can seek to any offset: consumer.seekToBeginning() replays all events from the start; consumer.seekToEnd() skips to the latest. This is what enables event replay — a powerful debugging and reprocessing capability unavailable in traditional queues.
Consumer Group
A set of consumers sharing a group ID that jointly consume a topic. Kafka assigns partitions across consumers in the group (each partition goes to exactly one consumer). If a consumer dies, Kafka rebalances: unassigned partitions are redistributed to surviving consumers. If you add consumers beyond the partition count, the extras sit idle — the maximum parallelism equals the partition count.
ISR (In-Sync Replicas)
The set of replicas that are fully caught up with the leader. A replica falls out of ISR if it falls too far behind (lag exceeds replica.lag.time.max.ms). The leader only acknowledges a message as committed when all ISR replicas have it (acks=all). Leader election on failure only considers ISR replicas — non-ISR replicas might be missing recent messages and could lose data if elected.
Retention Policy
Kafka stores messages for a configurable time (retention.ms, default 7 days) or size (retention.bytes). After that, old segments are deleted. Log compaction is an alternative: instead of time-based deletion, Kafka keeps only the latest message per key — useful for changelog topics where you want the current state (e.g., user preferences). Consumers that fall behind retention lose the ability to replay those messages.
Kafka Streams
A client library for building real-time stream processing applications on top of Kafka. Read from input topics, apply transformations (filter, map, join, aggregate), write to output topics — all within a Java application. Stateful operations (counts, joins) store state in embedded RocksDB, replicated via changelog topics for fault tolerance. Alternative to Flink or Spark Streaming for simpler, lower-overhead stream processing.
Key Facts
- LinkedIn built Kafka in 2010 to handle 1 trillion messages per day. The core insight: model the event bus as an append-only distributed log, not a queue. This made Kafka orders of magnitude faster than JMS/AMQP brokers of the time.
- Kafka achieves high throughput through batching and sequential I/O. Producers batch messages; brokers write in large sequential disk writes (fast even on spinning disks); consumers read sequentially. Sequential disk access can be faster than random memory access at scale.
- ZooKeeper was historically required for Kafka metadata and leader election. Kafka 3.x introduced KRaft mode (Kafka Raft Metadata mode) — ZooKeeper is no longer needed. KRaft stores metadata in a Kafka topic itself, simplifying operations significantly.
- The producer's send() is asynchronous by default. Calling send() puts the message in an internal buffer; a background sender thread batches and compresses messages and sends them to the broker. Use Futures or callbacks to get confirmation of delivery. Synchronous send (blocking on future.get()) eliminates batching benefits.
- Kafka Connect is a framework for streaming data between Kafka and external systems (databases, S3, Elasticsearch) using pre-built connectors. Debezium is a popular Kafka Connect source connector that reads MySQL/PostgreSQL/MongoDB change logs (CDC) and streams every database change as a Kafka event.
Real-World Applications
Microservice event bus
Services publish domain events (OrderPlaced, UserRegistered) to Kafka topics instead of calling each other directly. Other services subscribe to relevant topics and react asynchronously. Services are decoupled: the order service doesn't know the inventory, notification, or analytics services exist. New subscribers can be added without changing producers. Event replay lets new services backfill historical data from day one.
Real-time analytics pipeline
Application servers publish click events, page views, and transactions to Kafka. A Kafka Streams or Flink job aggregates events in real time (counts per minute, error rates, funnel conversion). Results are written to a fast read store (Redis, ClickHouse, Elasticsearch) for dashboards. The raw events in Kafka are retained for 7+ days for ad-hoc reprocessing and A/B analysis.
Change Data Capture (CDC)
Debezium reads the MySQL binary log and publishes every INSERT, UPDATE, DELETE as a Kafka event. Downstream services (search index, cache, data warehouse) react to database changes in near real time without polling. The database is the system of truth; Kafka propagates changes. Eliminates dual writes (writing to both DB and cache directly) and their consistency problems.
Frequently Asked Questions
What is the difference between Kafka and RabbitMQ?
RabbitMQ is a traditional message broker: messages are routed by the broker to queues, consumed by one consumer, and deleted after delivery. Good for task queues, RPC, and complex routing. Kafka is a distributed log: messages are appended to partitions, retained for a time period, and can be read by any number of consumer groups independently. Good for event streaming, data pipelines, and audit logs. Choose RabbitMQ for point-to-point task distribution; choose Kafka for event sourcing, fan-out to multiple consumers, or replay.
What happens if a consumer is slower than the producer?
Kafka handles this naturally: consumers read from their last committed offset. A slow consumer simply falls behind, accumulating lag (the gap between the latest offset and the consumer's offset). Kafka retains messages until retention.ms expires, giving the consumer time to catch up. You can monitor consumer lag with kafka-consumer-groups.sh --describe or tools like Burrow, Prometheus JMX exporter. If lag keeps growing, add more consumers (up to partition count) to increase throughput.
How many partitions should a topic have?
It depends on throughput and consumer parallelism. More partitions = more parallelism = higher throughput, but also more open file handles, more replication overhead, and longer leader election time on failure. A common rule of thumb: (target throughput) / (throughput per partition) for the number of partitions. If you need 12 parallel consumers, you need at least 12 partitions. Start conservative (6-12) and increase as needed — you can add partitions later, but existing messages won't be redistributed.
How do you guarantee message ordering in Kafka?
Ordering is guaranteed within a partition. If global ordering matters across all messages in a topic, use a single partition — but this limits throughput and parallelism. If ordering matters per entity (e.g., all events for a given user must be in order), use the user ID as the message key. Kafka hashes the key to a partition, so all messages for the same user go to the same partition in the same order. You get per-user ordering with full parallelism across users.