Consumer Offsets
BeginnerConsumer groups commit offsets to the __consumer_offsets internal topic; on restart or rebalance, the consumer resumes from the last committed offset.
Overview
An **offset** is the position of a record within a partition — a monotonically increasing integer starting at 0. Each **consumer group** tracks its own offset per partition, stored in the internal `__consumer_offsets` topic. When a consumer (re)starts, it fetches the last committed offset and resumes from there. **Committing an offset** means telling the broker "I have successfully processed all records up to and including this offset." If the consumer crashes before committing, the records will be redelivered after restart — this is the foundation of Kafka's at-least-once delivery guarantee. `auto.offset.reset` controls what happens when there is no committed offset (new group or offset expired): `earliest` (read from the beginning), `latest` (read only new records).
Auto-Commit vs Manual Commit
With `enable.auto.commit=true` (default), Kafka commits the current offset every `auto.commit.interval.ms` (default 5 s) in the background. This is simple but risky: records are committed before processing completes — a crash after commit but before processing means data loss. For at-least-once guarantees, use **manual commit** (`enable.auto.commit=false`) and commit only after successful processing.
# Auto-commit (default) — simple but can lose messages
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=5000 # commit every 5s
# Manual commit — at-least-once guarantee
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.listener.ack-mode=manual_immediate # Spring Kafka ack mode
@KafkaListener(topics = "orders")
public void process(Order order, Acknowledgment ack) {
try {
orderService.process(order);
ack.acknowledge(); // commit offset AFTER successful processing
} catch (RecoverableException e) {
// Don't ack — record will be redelivered on restart/rebalance
throw e;
}
}
# Spring Kafka ack modes:
# RECORD — commit after each record
# BATCH — commit after each batch returned by poll()
# MANUAL — commit when Acknowledgment.acknowledge() is called on batch end
# MANUAL_IMMEDIATE — commit synchronously right when acknowledge() is calledauto.offset.reset and Consumer Group Reset
`auto.offset.reset` applies when a consumer group has no committed offset (brand new group) or the committed offset is out of range (expired topic retention). `earliest` reads all available records; `latest` skips historical records and reads only new ones. To manually reset a consumer group's offset (re-process events), use `kafka-consumer-groups.sh --reset-offsets`.
# auto.offset.reset options
spring.kafka.consumer.auto-offset-reset=earliest # new group reads from partition start
spring.kafka.consumer.auto-offset-reset=latest # new group reads from now (default)
# Inspect current consumer group offsets
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group order-processor --describe
# Output:
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
# orders 0 1000 1050 50 worker-1-...
# orders 1 980 990 10 worker-2-...
# Reset offsets — re-process last 2 hours of events
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group order-processor --topic orders --reset-offsets --to-datetime 2025-03-22T10:00:00.000 --execute
# Reset to earliest — replay everything
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group order-processor --topic orders --reset-offsets --to-earliest --executeOffset Commit Strategies and At-Least-Once vs Exactly-Once
The three delivery semantics correspond to when and how offsets are committed relative to processing. At-most-once: commit before processing (may lose messages on crash). At-least-once: commit after processing (may duplicate on crash before commit). Exactly-once: atomic write + offset commit using Kafka transactions (`sendOffsetsToTransaction`). For most applications, at-least-once + idempotent processing is the practical target.
// At-most-once: commit offset BEFORE processing (default auto-commit)
// Risk: crash after commit, before processing → message lost
poll() → commit() → process()
// At-least-once: commit offset AFTER processing (manual commit)
// Risk: crash after process(), before commit() → redelivery + duplicate processing
poll() → process() → commit()
// Mitigate: design processing to be idempotent (check if already processed)
// Exactly-once: Kafka transactions (atomic process + offset commit)
producer.beginTransaction();
producer.send("output-topic", result);
producer.sendOffsetsToTransaction(offsets, consumerGroupMetadata);
producer.commitTransaction();
// Risk: complex, performance overhead, requires read_committed consumer isolation
// Practical recommendation for most services:
// Use at-least-once + idempotency key check:
if (processedRepo.existsByMessageId(record.key())) {
ack.acknowledge(); // duplicate — skip safely
return;
}
processOrder(record.value());
processedRepo.save(new ProcessedMessage(record.key()));
ack.acknowledge();Key Points to Remember
- 1Offsets are committed per consumer group per partition to __consumer_offsets internal topic
- 2Auto-commit can lose messages: offset committed before processing completes
- 3Manual commit (MANUAL_IMMEDIATE) ensures offset committed only after successful processing
- 4auto.offset.reset=earliest: start from beginning; latest: start from now (for new groups)
- 5Reset offsets with kafka-consumer-groups.sh --reset-offsets to replay historical events
- 6At-least-once + idempotent processing is the recommended pattern for most services
Interview Questions
Sign in to ask AriaWhat is the risk of using enable.auto.commit=true in a Kafka consumer?
What does auto.offset.reset=earliest do and when does it take effect?
How would you re-process the last 2 hours of Kafka events for a consumer group?
Explain the three delivery semantics (at-most-once, at-least-once, exactly-once) in terms of offset commit timing.
What is consumer lag and how would you alert on it?
Ask Aria about Consumer Offsets
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.