Home/Learn/Apache Kafka/Offsets & Log Structure

Offsets & Log Structure

Beginner

Fundamentals

Each message in a partition has a monotonically increasing offset; the log is append-only and immutable, enabling deterministic replay and position-based consumption.

Overview

Every message written to a Kafka partition is assigned a unique, monotonically increasing 64-bit integer called an offset. The partition log is a sequence of immutable segments on disk. Consumers track their position using offsets stored in the internal __consumer_offsets topic. Committing an offset means "I have processed everything up to this point." Kafka supports three delivery semantics: at-most-once (commit before processing), at-least-once (commit after processing), and exactly-once (idempotent producer + transactional consumer).

Log Structure & Segments

Each partition is stored as a series of segment files on disk. A segment becomes immutable once it reaches log.segment.bytes (default 1 GB) or log.roll.ms. Older segments are deleted or compacted based on retention policy.

Shell — partition log structure on disk

# On-disk layout of a partition (orders-0)
/var/kafka/data/orders-0/
  00000000000000000000.log       # messages at offsets 0–999999
  00000000000000000000.index     # offset → physical file position
  00000000000000000000.timeindex # timestamp → offset lookup
  00000000000001000000.log       # messages at offsets 1000000–1999999
  00000000000001000000.index
  00000000000002000000.log       # active (current) segment

# Each .log entry: [offset][message size][CRC][magic][attributes][timestamp][key][value]

# Retention settings (application.properties / topic config)
# Log retention — delete segments older than 7 days
log.retention.hours=168

# Or size-based retention — keep last 10 GB per partition
log.retention.bytes=10737418240

# Log compaction — keep only the latest record per key
log.cleanup.policy=compact    # useful for changelog topics (state stores)

Consumer Offset Management

Consumer groups store committed offsets in the __consumer_offsets topic. Spring Kafka can commit synchronously (safe) or asynchronously (faster). auto-commit polls on a schedule but risks duplicates; manual commit gives full control.

Properties + Java + Shell — offset commit strategies

# application.properties — offset commit modes

# Auto-commit (default) — commits every auto.commit.interval.ms
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=5000   # ms

# Manual commit — full control, preferred for at-least-once
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.listener.ack-mode=manual_immediate

# Spring AMQP manual ack
@KafkaListener(topics = "orders")
public void process(OrderEvent event, Acknowledgment ack) {
    try {
        orderService.handle(event);
        ack.acknowledge();   // commit after successful processing
    } catch (Exception e) {
        // don't ack → record will be redelivered after poll timeout
        log.error("Processing failed", e);
    }
}

# Reset consumer group offset (CLI)
kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group order-processor \
  --topic orders \
  --reset-offsets --to-earliest --execute

Delivery Semantics

Delivery semantics describe what happens if a broker or consumer fails mid-processing. At-least-once is the practical default. Exactly-once requires idempotent producers and transactional consumers.

Java + Properties — delivery semantics

// At-most-once — commit before processing (risk: message lost on crash)
@KafkaListener(topics = "orders")
public void processAtMostOnce(OrderEvent event, Acknowledgment ack) {
    ack.acknowledge();          // ✗ commit first
    orderService.handle(event); // crash here → message lost
}

// At-least-once — commit after processing (risk: duplicate on crash)
@KafkaListener(topics = "orders")
public void processAtLeastOnce(OrderEvent event, Acknowledgment ack) {
    orderService.handle(event); // process first
    ack.acknowledge();          // ✓ commit after — crash → reprocess (duplicate)
}

// Exactly-once — idempotent producer + transactional consumer
# Producer config
spring.kafka.producer.properties.enable.idempotence=true
spring.kafka.producer.properties.transactional.id=order-producer-1

# Consumer config (only reads committed messages from transactional producers)
spring.kafka.consumer.isolation-level=read_committed

Key Points to Remember

1Offset is a monotonically increasing integer identifying a message's position in a partition.
2The partition log is append-only and immutable; old segments are deleted or compacted.
3Consumers store committed offsets in __consumer_offsets; committing means "processed up to here".
4auto.offset.reset=earliest replays from the beginning; latest skips existing messages.
5At-least-once is the practical default: commit after processing, deduplicate downstream.
6Exactly-once requires enable.idempotence=true on producer + read_committed isolation on consumer.

Interview Questions

What is a Kafka offset and how is it used by consumers?

EasyTCS

What is the difference between at-least-once and exactly-once delivery in Kafka?

MediumAmazon

What happens if a consumer commits its offset before processing the message?

MediumFlipkart

What is log compaction and when would you use it?

HardLinkedIn

How do you reset a consumer group to re-process messages from the beginning?

MediumInfosys

Ask Aria about Offsets & Log Structure

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Leaders & Followers

Kafka Producer Basics