Kafka Cluster Scaling
AdvancedAdd brokers and use partition reassignment to rebalance leadership; Kafka 3.x's Cruise Control automates load balancing based on disk, CPU, and network metrics.
Overview
Scaling a Kafka cluster involves adding brokers (horizontal scaling) and rebalancing partition leadership across the expanded cluster. Kafka does not auto-rebalance when new brokers are added — you must explicitly reassign partitions using kafka-reassign-partitions.sh or LinkedIn's Cruise Control. Partition count is the primary throughput lever: more partitions = more consumer parallelism. However, adding partitions to an existing topic is irreversible and breaks key ordering guarantees if key-based routing is used. Cluster scaling must be coordinated carefully to avoid overwhelming brokers during data movement.
Adding brokers and partition reassignment
Adding a broker does not automatically move any data. Use kafka-reassign-partitions.sh to generate and execute a reassignment plan. The plan specifies which partitions move to which brokers. Throttle the reassignment to limit inter-broker replication traffic and avoid starving producer/consumer workloads.
# Step 1: Generate reassignment plan for a topic
cat > topics-to-move.json <<EOF
{"topics": [{"topic": "orders"}], "version": 1}
EOF
kafka-reassign-partitions.sh \
--bootstrap-server broker:9092 \
--broker-list "1,2,3,4" ← new broker 4 is added \
--topics-to-move-json-file topics-to-move.json \
--generate > reassignment-plan.json
# Step 2: Execute with throttle (limit to 50 MB/s to avoid impacting live traffic)
kafka-reassign-partitions.sh \
--bootstrap-server broker:9092 \
--reassignment-json-file reassignment-plan.json \
--execute \
--throttle 52428800 # 50 MB/s in bytes
# Step 3: Monitor progress
kafka-reassign-partitions.sh \
--bootstrap-server broker:9092 \
--reassignment-json-file reassignment-plan.json \
--verify
# Step 4: Remove throttle after reassignment completes
kafka-configs.sh --bootstrap-server broker:9092 \
--entity-type brokers --entity-default \
--alter --delete-config leader.replication.throttled.rateIncreasing partition count and preferred leader election
Increase partitions on a topic when producer throughput or consumer parallelism is bottlenecked. Partition increase is irreversible and breaks key-based ordering — all keys are re-distributed to new partitions. After reassignment or broker restarts, preferred leader election restores balanced leadership to the original preferred (first) replica.
# Increase partition count (irreversible)
kafka-topics.sh --bootstrap-server broker:9092 \
--alter --topic orders \
--partitions 24 # was 12 — increase to 24
# WARNING: key-based ordering breaks for existing keys after partition count change
# Records with the same key may now hash to different partitions
# Check current partition count
kafka-topics.sh --bootstrap-server broker:9092 \
--describe --topic orders
# Preferred leader election — restore balanced leadership after restart/reassignment
kafka-leader-election.sh \
--bootstrap-server broker:9092 \
--election-type PREFERRED \
--all-topic-partitions
# Or for specific topic
kafka-leader-election.sh \
--bootstrap-server broker:9092 \
--election-type PREFERRED \
--topic orders
# Monitor leader imbalance
kafka-topics.sh --bootstrap-server broker:9092 \
--describe | grep "Leader: " | awk '{print $4}' | sort | uniq -c
# Shows leader distribution across brokers — should be balancedCruise Control for automated rebalancing
LinkedIn's Cruise Control monitors broker resource utilisation (disk, CPU, network, partition count) and automatically generates optimal reassignment plans to balance the cluster. It exposes a REST API for triggering rebalances, adding/removing brokers, and monitoring cluster state. In Kafka 3.x deployments on Kubernetes (Strimzi operator), Cruise Control is integrated as a first-class feature.
# Cruise Control REST API — trigger automated rebalance
# (runs as a separate service alongside Kafka brokers)
# Get cluster load overview
curl http://cruise-control:9090/kafkacruisecontrol/load
# Generate rebalance proposal (dry run)
curl -X POST http://cruise-control:9090/kafkacruisecontrol/rebalance \
-d "dry_run=true&goals=NetworkInboundCapacityGoal,NetworkOutboundCapacityGoal,DiskCapacityGoal"
# Execute rebalance
curl -X POST http://cruise-control:9090/kafkacruisecontrol/rebalance \
-d "dry_run=false&allow_capacity_estimation=true"
# Add broker (decommission is also supported)
curl -X POST http://cruise-control:9090/kafkacruisecontrol/add_broker \
-d "brokerid=4&dry_run=false"
# Strimzi KafkaRebalance (Kubernetes operator integration)
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
name: cluster-rebalance
labels:
strimzi.io/cluster: my-kafka
spec:
goals:
- NetworkInboundCapacityGoal
- NetworkOutboundCapacityGoal
- DiskCapacityGoalKey Points to Remember
- 1Adding brokers does not auto-rebalance — use kafka-reassign-partitions.sh or Cruise Control to move partitions
- 2Always throttle partition reassignment to prevent overwhelming brokers and impacting live traffic
- 3Partition count increase is irreversible and breaks key ordering — plan partition counts upfront
- 4Preferred leader election restores balanced leadership after broker restarts or completed reassignments
- 5Cruise Control automates rebalancing decisions based on disk, CPU, and network utilisation metrics
- 6In Strimzi (Kubernetes), Cruise Control is integrated — trigger rebalances via KafkaRebalance CRDs
Interview Questions
Sign in to ask AriaWhen you add a new broker to a Kafka cluster, what happens to existing partitions?
Why should you throttle partition reassignment and what metric controls the throttle rate?
What are the consequences of increasing a topic's partition count on a key-partitioned topic?
What is preferred leader election and when does it need to be triggered?
How does Cruise Control decide which partitions to move when rebalancing a cluster?
Ask Aria about Kafka Cluster Scaling
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.