Consumer Rebalancing
IntermediateRebalancing redistributes partition assignments among consumers in a group. It is triggered by group membership changes and can cause brief processing pauses.
Overview
During an eager (stop-the-world) rebalance, all consumers revoke their partitions and the coordinator reassigns. Cooperative/incremental rebalancing (default since Kafka 2.4) only revokes affected partitions, minimising downtime.
Eager vs Cooperative Rebalancing
Cooperative Sticky Assignor revokes only the partitions that must move, letting other consumers continue processing.
// Use Cooperative Sticky — best for most use cases
spring.kafka.consumer.properties.partition.assignment.strategy=\
org.apache.kafka.clients.consumer.CooperativeStickyAssignor
@Component
public class RebalanceListener implements ConsumerRebalanceListener {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
consumer.commitSync(currentOffsets(partitions)); // commit before revoke
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
log.info("Assigned: {}", partitions);
}
}Tuning to Reduce Rebalances
max.poll.interval.ms is the most common cause of unexpected rebalances — if processing takes longer than this, the consumer is kicked out.
# Rebalance tuning
session.timeout.ms=45000
heartbeat.interval.ms=3000 # < session.timeout.ms / 3
max.poll.interval.ms=300000 # set to max expected processing time
max.poll.records=500 # reduce if processing is slowKey Points to Remember
- 1Eager rebalance: ALL consumers pause — all partitions revoked and reassigned
- 2Cooperative Sticky: only affected partitions move, others continue
- 3max.poll.interval.ms exceeded → consumer kicked from the group
- 4Implement ConsumerRebalanceListener to commit offsets before revocation
- 5Static group membership (group.instance.id) eliminates rebalances on rolling restarts
Interview Questions
Sign in to ask AriaWhat triggers a Kafka consumer group rebalance?
What is the difference between eager and cooperative rebalancing in Kafka?
Why does max.poll.interval.ms cause a consumer to be kicked from the group?
How does static group membership (group.instance.id) reduce rebalances?
How would you safely commit offsets when partitions are being revoked during a rebalance?
Ask Aria about Consumer Rebalancing
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.