ZooKeeper & KRaft Mode
IntermediateZooKeeper stored Kafka metadata historically; KRaft (Kafka Raft, 3.x) replaces it with an internal quorum controller, eliminating the external dependency and simplifying operations.
Overview
Historically, Kafka depended on Apache ZooKeeper to store cluster metadata: broker registrations, topic configurations, partition leadership, and ACLs. ZooKeeper was a separate cluster to manage, a separate failure domain, and a bottleneck for large clusters (ZooKeeper could not handle millions of partitions efficiently). KRaft (Kafka Raft Metadata, KIP-500) replaces ZooKeeper with an internal Raft-based quorum of controller nodes. KRaft is production-ready since Kafka 3.3 and is the only option in Kafka 4.0. Benefits: single binary, faster startup, faster controller failover, and support for millions of partitions.
ZooKeeper-Based Architecture (Legacy)
In the ZooKeeper model, a separate ZooKeeper quorum (typically 3 or 5 nodes) stores all Kafka metadata. The Kafka controller broker watches ZooKeeper for changes and coordinates partition leadership elections.
# Legacy ZooKeeper-based broker config (server.properties)
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka # ZK quorum + chroot
broker.id=1
log.dirs=/var/kafka/data
# Start ZooKeeper first
bin/zookeeper-server-start.sh config/zookeeper.properties
# Then start Kafka brokers
bin/kafka-server-start.sh config/server.properties
# View metadata stored in ZooKeeper
bin/zookeeper-shell.sh zk1:2181
# > ls /kafka/brokers/ids → active broker IDs
# > get /kafka/controller → current controller broker
# > ls /kafka/topics → all topics
# Limitations:
# - ZK handles ~200k partition leadership changes
# - Slow controller failover (30-120 seconds in large clusters)
# - Additional ops burden: separate ZK JVM, separate monitoringKRaft Mode Architecture
In KRaft, a subset of Kafka brokers act as controller nodes using the Raft consensus algorithm. All cluster metadata is stored in an internal topic (__cluster_metadata). The active controller handles metadata operations; followers replicate via Raft.
# KRaft combined mode (controller + broker on same node — dev/small clusters)
# server.properties
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@node1:9093,2@node2:9093,3@node3:9093
listeners=PLAINTEXT://node1:9092,CONTROLLER://node1:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/var/kafka/data
# Format storage (run ONCE before first start)
CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)
bin/kafka-storage.sh format \
--config config/server.properties \
--cluster-id $CLUSTER_ID
# Start broker
bin/kafka-server-start.sh config/server.properties
# Separate controller and broker roles (production)
# node1,2,3: process.roles=controller (dedicated controller quorum)
# node4..N: process.roles=broker (dedicated broker nodes)Migrating from ZooKeeper to KRaft
Kafka 3.x supports online migration from ZooKeeper to KRaft mode. The migration process uses a dedicated migration tool. After migration, ZooKeeper can be decommissioned. Kafka 4.0 drops ZooKeeper support entirely.
# Kafka 3.5+ online migration steps (high level)
# Step 1 — Deploy KRaft controller quorum alongside existing ZK-based cluster
# Set migration.enabled=true and zookeeper.connect in KRaft controller config
# Step 2 — Roll existing brokers to "migration mode"
# server.properties for each broker:
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
migration.enabled=true
# Step 3 — Verify migration in logs
# "Completed migration of metadata from ZooKeeper to KRaft"
# Step 4 — Remove ZooKeeper config from brokers and restart
# Remove: zookeeper.connect, migration.enabled
# Step 5 — Decommission ZooKeeper cluster
# Verify cluster is running in KRaft mode
bin/kafka-metadata-quorum.sh \
--bootstrap-server localhost:9092 \
describe --statusKey Points to Remember
- 1ZooKeeper stored Kafka metadata externally — a separate cluster to manage and monitor.
- 2KRaft replaces ZooKeeper with an internal Raft quorum stored in __cluster_metadata topic.
- 3KRaft is GA since Kafka 3.3; ZooKeeper mode is removed in Kafka 4.0.
- 4KRaft supports millions of partitions vs ~200k with ZooKeeper.
- 5Controller failover with KRaft is milliseconds vs 30-120 seconds with ZooKeeper.
- 6kafka-storage.sh format must be run once before starting a KRaft cluster.
Interview Questions
Sign in to ask AriaWhat was ZooKeeper's role in Kafka and why is it being replaced?
What is KRaft mode and how does it change Kafka's architecture?
What are the benefits of KRaft over ZooKeeper-based Kafka?
How does the Raft consensus algorithm work in the context of KRaft?
What is the process for migrating a ZooKeeper-based Kafka cluster to KRaft?
Ask Aria about ZooKeeper & KRaft Mode
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.