Consumer Rebalancing
IntermediateWhen a consumer joins or leaves a group, the broker-coordinated rebalance reassigns partitions; use Cooperative Sticky Assignor (incremental rebalance) to minimise disruption.
Overview
A **rebalance** is triggered when the composition of a consumer group changes: a consumer joins (new pod, restart), leaves (crash, scale-down), or fails to send a heartbeat within `session.timeout.ms`. During a **stop-the-world (eager) rebalance**, all consumers revoke their partitions, the Group Coordinator assigns new ownership, and consumption pauses for the entire group — even consumers whose partitions didn't change. This rebalance storm causes consumer lag spikes and duplicate processing (messages re-processed from the last committed offset). The **Cooperative Sticky Assignor** (default since Kafka 3.1 in Spring Boot) uses incremental rebalances: only the partitions that need to move are revoked, and the rest keep consuming uninterrupted.
Rebalance Triggers and the Stop-the-World Problem
The Group Coordinator detects consumer failure when it misses a heartbeat within `session.timeout.ms`. A long-running `poll()` loop that exceeds `max.poll.interval.ms` also triggers a rebalance (the consumer is considered "dead"). During an eager rebalance, all partitions are revoked from all consumers simultaneously — causing a processing gap.
# Key timeouts — tuned to reduce spurious rebalances
session.timeout.ms=45000 # max time broker waits for heartbeat (default: 45s)
heartbeat.interval.ms=3000 # how often consumer sends heartbeat (should be <1/3 session)
max.poll.interval.ms=300000 # max time between poll() calls before consumer is dead
# Spring Kafka equivalent
spring.kafka.consumer.properties.session.timeout.ms=45000
spring.kafka.consumer.properties.max.poll.interval.ms=300000
spring.kafka.listener.poll-timeout=3000
# Reduce max.poll.records if processing is slow — prevents exceeding max.poll.interval.ms
spring.kafka.consumer.max-poll-records=100 # default 500
# Warning signs of rebalance thrashing:
# - Logs: "Attempt to heartbeat failed since group is rebalancing"
# - Consumer lag spikes at regular intervals
# - Duplicate records processed (offset re-committed after rebalance)Cooperative Sticky Assignor — Incremental Rebalance
The **Cooperative Sticky Assignor** (or `CooperativeStickyAssignor`) performs rebalances in two rounds: first it determines which partitions need to move, then only those consumers revoke their partitions and rejoin. Consumers keeping their partitions never stop. This turns a stop-the-world pause into a brief, targeted reassignment.
# Enable Cooperative Sticky Assignor (default in Kafka 3.x / Spring Boot 3)
spring.kafka.consumer.properties.partition.assignment.strategy=\
org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# Java producer config
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
# Comparison:
# Eager (RangeAssignor / RoundRobinAssignor):
# - All consumers revoke ALL partitions → full pause
# - Simple, well-understood, works in all Kafka versions
#
# Cooperative Sticky (default from Kafka 3.1):
# - Only moved partitions are revoked → minimal pause
# - 2 rounds of rebalance (slightly more complex)
# - "Sticky" = tries to keep same consumer-partition assignments
#
# Migration from Eager to Cooperative: requires a rolling restart with
# intermediate strategy: [CooperativeStickyAssignor, RangeAssignor]ConsumerRebalanceListener — Commit Before Revoke
When using manual offset commits, implement `ConsumerRebalanceListener` to commit offsets for partitions being revoked before they are reassigned. Without this, the new owner re-processes messages from the last committed offset — causing duplicates. `onPartitionsRevoked()` is your last chance to commit.
@Component
class OrderConsumer implements ConsumerRebalanceListener {
@Autowired KafkaTemplate<String, ?> template;
private final Map<TopicPartition, OffsetAndMetadata> currentOffsets = new HashMap<>();
@KafkaListener(topics = "orders")
public void consume(ConsumerRecord<String, Order> record,
Acknowledgment ack,
Consumer<?, ?> consumer) {
process(record.value());
// Track latest offset per partition
currentOffsets.put(
new TopicPartition(record.topic(), record.partition()),
new OffsetAndMetadata(record.offset() + 1));
ack.acknowledge();
}
// Called BEFORE partitions are revoked — commit pending offsets
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
consumer.commitSync(currentOffsets); // sync commit — blocks until done
partitions.forEach(currentOffsets::remove);
log.info("Committed offsets before rebalance for partitions: {}", partitions);
}
// Called AFTER new partitions are assigned — reset state
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
log.info("New partitions assigned: {}", partitions);
}
}Key Points to Remember
- 1Rebalance triggered by: consumer join/leave, heartbeat timeout, max.poll.interval.ms exceeded
- 2Eager rebalance: all consumers pause and revoke all partitions — causes processing gaps
- 3Cooperative Sticky Assignor: only revoked partitions pause; rest continue — default in Kafka 3.x
- 4Tune max.poll.records to prevent exceeding max.poll.interval.ms on slow processing
- 5ConsumerRebalanceListener.onPartitionsRevoked(): commit offsets before revocation to avoid duplicates
- 6heartbeat.interval.ms should be < 1/3 of session.timeout.ms
Interview Questions
Sign in to ask AriaWhat triggers a Kafka consumer group rebalance?
What is the difference between eager and cooperative rebalancing in Kafka?
Why does exceeding max.poll.interval.ms trigger a rebalance?
How does ConsumerRebalanceListener help prevent duplicate processing on rebalance?
What is the Cooperative Sticky Assignor and why is it preferred over the default Range Assignor?
Ask Aria about Consumer Rebalancing
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.