Consumer Groups
BeginnerConsumers in the same group cooperatively consume partitions; each partition is assigned to exactly one consumer, enabling parallel horizontal scaling up to the partition count.
Overview
Consumer groups are Kafka's mechanism for parallel, scalable, and fault-tolerant message consumption. Every consumer belongs to a named group (group.id). Kafka ensures that each partition of a topic is consumed by exactly one consumer within a group at any given time — this is the fundamental guarantee that prevents duplicate processing within the group. Multiple consumer groups can read the same topic independently, each maintaining its own offset, which is how Kafka supports pub-sub fan-out: add a new consumer group and it gets a full copy of all events without affecting existing consumers. If a consumer crashes, Kafka rebalances the group and assigns that consumer's partitions to the remaining members — making the system self-healing.
Partition Assignment & Parallelism
Kafka distributes partitions across consumers in a group using an assignor strategy. With 12 partitions and 3 consumers, each consumer gets 4 partitions. The upper bound on parallelism is the partition count — adding a 13th consumer to a 12-partition topic leaves one consumer idle.
Conversely, if you have fewer consumers than partitions, each consumer handles multiple partitions. If a consumer in the group crashes, the coordinator triggers a rebalance and redistributes its partitions to surviving members, resuming from the last committed offset.
// Spring Boot — @KafkaListener with consumer group
@Component
public class OrderConsumer {
// All instances of this service share group "order-service"
// Kafka assigns partitions across them automatically
@KafkaListener(
topics = "orders",
groupId = "order-service",
concurrency = "3" // 3 listener threads per instance
)
public void consume(ConsumerRecord<String, String> record) {
log.info("Partition={} Offset={} Key={} Value={}",
record.partition(), record.offset(),
record.key(), record.value());
processOrder(record.value());
}
}
// application.yml
spring:
kafka:
consumer:
group-id: order-service
auto-offset-reset: earliest # start from beginning if no committed offset
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializerMultiple Consumer Groups — Fan-Out
Different consumer groups are completely independent. Each group has its own committed offsets and reads the topic from its own position. This enables multiple applications to process the same stream of events for different purposes — an order-service processes orders, a notification-service sends emails, and an analytics-service writes to a data warehouse — all from the same Kafka topic without interfering with each other.
// Topic: "orders" is consumed by 3 independent groups
// Group 1: order-processing-service → fulfills orders
// Group 2: notification-service → sends confirmation emails
// Group 3: analytics-service → writes to data warehouse
// List consumer groups and their lag
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-service
# Output:
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
# order-processing-service orders 0 1024 1024 0
# order-processing-service orders 1 1019 1021 2
# order-processing-service orders 2 987 987 0Rebalancing & Partition Assignment Strategies
When a consumer joins or leaves a group, a rebalance is triggered. During a rebalance, all consumers in the group stop processing (stop-the-world). The group coordinator (a Kafka broker) facilitates the rebalance using an assignor strategy:
RangeAssignor (default) — assigns contiguous partitions to consumers. Can be uneven. RoundRobinAssignor — distributes partitions evenly in round-robin order. StickyAssignor — minimises partition movement on rebalance (preserves existing assignments where possible). CooperativeStickyAssignor (recommended) — incremental rebalance, consumers revoke only the partitions they must give up, so unaffected consumers keep processing during the rebalance.
# Configure the Cooperative Sticky Assignor to reduce rebalance downtime
spring:
kafka:
consumer:
group-id: order-service
properties:
partition.assignment.strategy: >
org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# Detect consumer group issues
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--list
# Reset offset to re-process from the beginning (use with care in production)
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--group order-service \
--topic orders \
--reset-offsets --to-earliest --executeKey Points to Remember
- 1Each partition is assigned to exactly one consumer within a group — this prevents duplicate processing within the group.
- 2The maximum parallelism for a consumer group equals the number of partitions; extra consumers sit idle.
- 3Multiple consumer groups read the same topic independently — each group gets its own copy of every event (pub-sub fan-out).
- 4If a consumer crashes, Kafka triggers a rebalance and reassigns its partitions to surviving group members.
- 5CooperativeStickyAssignor performs incremental rebalances so unaffected consumers keep processing during partition handoff.
- 6Monitor consumer lag (LOG-END-OFFSET minus CURRENT-OFFSET) per partition — high lag means the consumer cannot keep up.
Interview Questions
Sign in to ask AriaWhat is a Kafka consumer group and how does partition assignment work within it?
A Kafka topic has 6 partitions. You start 8 consumers in the same group. What happens to the extra 2 consumers?
How do multiple consumer groups consuming the same topic affect each other?
What triggers a consumer group rebalance and what is its impact on message processing?
What is the difference between RangeAssignor and CooperativeStickyAssignor? Which would you use in production?
Ask Aria about Consumer Groups
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.