Auto vs Manual Offset Commit
Intermediateenable.auto.commit=true commits periodically and risks data loss; manual commitSync/commitAsync after processing guarantees at-least-once but requires idempotent consumers.
Overview
Kafka consumers track their progress by committing offsets to the `__consumer_offsets` topic. When a consumer restarts or a rebalance occurs, it resumes from the last committed offset. Two commit modes exist: auto-commit (the default, enable.auto.commit=true) periodically commits the latest polled offset on a timer — simple but unsafe if the consumer crashes between poll and commit. Manual commit gives the application full control over when to commit: only after the records have been successfully processed, giving at-least-once semantics. Spring Kafka wraps this with a clean acknowledgement API. The key tradeoff: auto-commit risks data loss; manual commit risks duplicates (safe with idempotent processing logic).
Auto-Commit — The Hidden Trap
With enable.auto.commit=true (default), Kafka commits the latest polled offset every auto.commit.interval.ms (default 5 000 ms). The timer fires independently of your processing. This creates a race:
1. poll() returns records [100..150] 2. Auto-commit fires → offset 150 committed 3. Consumer crashes while processing record 120 4. On restart, consumer starts from 151 — records 120–150 are lost forever
Auto-commit is only safe for use cases where data loss is acceptable (e.g., non-critical metrics dashboards). For anything else, disable it and commit manually.
// application.yml — auto-commit (default, NOT recommended for production)
spring:
kafka:
consumer:
enable-auto-commit: true
auto-commit-interval: 5000 # commit every 5 seconds regardless of processing
auto-offset-reset: earliest
// The danger in code
@KafkaListener(topics = "payments")
public void process(List<ConsumerRecord<String, Payment>> records) {
for (ConsumerRecord<String, Payment> rec : records) {
// If we crash here, offsets may already be committed — DATA LOSS!
paymentService.handle(rec.value());
}
}Manual Commit — commitSync vs commitAsync
With enable.auto.commit=false, the application decides when to commit. Two APIs exist:
**commitSync()** — blocks until the broker acknowledges the commit. Reliable but adds latency. Use in error handlers or when correctness is critical.
**commitAsync(callback)** — fire-and-forget with an optional callback. Higher throughput but if the commit fails (transient network error), it is not automatically retried — you must handle failures in the callback. A common pattern: use commitAsync() in the normal processing loop and commitSync() in the finally block or shutdown hook.
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); // ← disable auto-commit
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("payments"));
try {
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
processPayment(record.value());
}
// Commit AFTER processing — at-least-once delivery
consumer.commitAsync((offsets, ex) -> {
if (ex != null) log.error("Commit failed for {}", offsets, ex);
});
}
} finally {
// Synchronous commit on shutdown for reliability
consumer.commitSync();
consumer.close();
}Spring Kafka Manual Acknowledgement
Spring Kafka's @KafkaListener wraps the low-level commit API. Set ack-mode to MANUAL or MANUAL_IMMEDIATE, inject Acknowledgment into the listener method, and call ack.acknowledge() when done. MANUAL_IMMEDIATE commits immediately on ack(); MANUAL batches commits to the next poll(). This is the idiomatic Spring way to get manual commit without managing a consumer loop yourself.
# application.yml
spring:
kafka:
consumer:
enable-auto-commit: false
listener:
ack-mode: MANUAL_IMMEDIATE # commit immediately on acknowledge()
// Listener
@Component
public class PaymentConsumer {
@KafkaListener(topics = "payments", groupId = "payment-service")
public void onPayment(
ConsumerRecord<String, Payment> record,
Acknowledgment ack) {
try {
paymentService.process(record.value());
ack.acknowledge(); // commit this offset
} catch (RecoverableException e) {
// Do NOT ack — message will be redelivered after rebalance/restart
log.warn("Transient error on {}, will retry", record.offset());
} catch (FatalException e) {
// Dead-letter or skip — still ack to avoid infinite retry
deadLetterPublisher.publish(record);
ack.acknowledge();
}
}
}Key Points to Remember
- 1Auto-commit (default) commits offsets on a timer, independently of processing — a crash between poll and commit causes data loss.
- 2Disable auto-commit and commit manually after successful processing for at-least-once delivery semantics.
- 3commitAsync() is higher throughput; commitSync() is reliable — use async in the loop, sync in the shutdown/finally block.
- 4Spring Kafka ack-mode: MANUAL_IMMEDIATE commits on ack.acknowledge(); MANUAL batches to the next poll().
- 5Manual commit with at-least-once means duplicates are possible on retry — consumers must be idempotent.
- 6Never commit offsets inside the catch block of a transient error — let the message be redelivered.
Interview Questions
Sign in to ask AriaWhat is the risk of using enable.auto.commit=true in a Kafka consumer?
What is the difference between commitSync and commitAsync?
How does Spring Kafka's ack-mode: MANUAL_IMMEDIATE work?
A consumer processes 100 records, commits offsets, then crashes. When it restarts, does it replay those 100 records?
How would you implement exactly-once processing at the application level without Kafka transactions?
Ask Aria about Auto vs Manual Offset Commit
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.