Brokers & Clusters
BeginnerA Kafka cluster is a group of broker nodes; topics are spread across brokers for fault tolerance and throughput, and a controller broker coordinates partition leadership.
Overview
A Kafka broker is a server process that stores topic partition data and serves producer and consumer requests. Multiple brokers form a cluster. Each topic partition has one leader broker (handles all reads and writes) and zero or more follower brokers (replicate data for fault tolerance). A controller broker — elected via ZooKeeper or the KRaft quorum — manages partition leadership elections and cluster membership. Horizontal scaling is achieved by adding brokers and reassigning partitions.
Broker Responsibilities
Each broker owns a set of partition replicas. For a given partition, the leader handles all produce and fetch requests while followers replicate in the background. If the leader fails, an in-sync replica (ISR) is elected as the new leader within seconds.
# View cluster metadata with kafka-topics.sh
kafka-topics.sh \
--bootstrap-server localhost:9092 \
--describe \
--topic orders
# Output (example):
# Topic: orders Partitions: 3 ReplicationFactor: 3
# Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
# Partition: 1 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
# Partition: 2 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
#
# Isr = in-sync replicas — only these can become leader on failoverReplication Factor & ISR
replication.factor controls how many broker copies a partition has. ISR (in-sync replicas) are followers that are caught up within replica.lag.time.max.ms. min.insync.replicas + acks=all together guarantee durability.
# Create topic with RF=3 and 6 partitions
kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create \
--topic payments \
--partitions 6 \
--replication-factor 3
# Producer durability settings (application.properties)
spring.kafka.producer.acks=all
spring.kafka.producer.properties.min.insync.replicas=2
# With acks=all and min.insync.replicas=2:
# The leader waits for 2 ISR members to confirm before ack-ing the producer
# → tolerates 1 broker failure without data lossController & KRaft
The controller broker watches for broker failures and triggers partition leader elections. In KRaft mode (Kafka 3.x), a built-in Raft quorum replaces ZooKeeper, reducing operational complexity and improving startup and failover time.
# KRaft mode broker server.properties
process.roles=broker,controller # combined mode (dev/small clusters)
node.id=1
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
log.dirs=/var/kafka/data
# Format storage (required once before first start)
kafka-storage.sh format \
--config /etc/kafka/server.properties \
--cluster-id $(kafka-storage.sh random-uuid)
# Verify controller status
kafka-metadata-shell.sh \
--snapshot /var/kafka/data/__cluster_metadata-0/00000000000000000000.logKey Points to Remember
- 1A broker is a Kafka server; a cluster is a group of cooperating brokers.
- 2Each partition has one leader broker and N−1 follower replicas.
- 3Only the leader handles reads and writes; followers replicate asynchronously.
- 4ISR (in-sync replicas) are followers within replica.lag.time.max.ms of the leader.
- 5acks=all + min.insync.replicas=2 is the standard durability configuration.
- 6KRaft (Kafka 3.x) replaces ZooKeeper with an internal Raft quorum controller.
Interview Questions
Sign in to ask AriaWhat is the role of the controller broker in a Kafka cluster?
What is an ISR and why does it matter for fault tolerance?
What happens to a partition when its leader broker goes down?
Explain acks=all and min.insync.replicas and how they interact.
What problem does KRaft mode solve compared to ZooKeeper-based Kafka?
Ask Aria about Brokers & Clusters
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.