Delivery Semantics — Cheat Sheet

At-least-once guarantees no message is lost; consumers may process duplicates after rebalance or restart, so consumer logic must be idempotent.

• ✓At-least-once: every record is processed at least once — duplicates can occur when a crash happens between processing and committing the offset.
• ✓Commit offsets AFTER processing (manual commit, enable.auto.commit=false) to guarantee at-least-once; committing before risks at-most-once (data loss).
• ✓Make consumer logic idempotent: use a processed_events deduplication table keyed by topic+partition+offset.
• ✓Producer-side duplicates: a producer retry after a lost ACK can also produce duplicates; enable idempotence=true to prevent this.
• ✓At-least-once is the pragmatic default for most systems; exactly-once is only necessary when duplicate side effects are unacceptable (e.g., charge a payment card).
• ✓Natural idempotency (INSERT ON DUPLICATE KEY, upserts) is preferable to deduplication tables where possible — simpler and faster.

// Safe at-least-once consumer — commit AFTER processing
Properties props = new Properties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);  // critical

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

    for (ConsumerRecord<String, String> r : records) {
        processOrder(r.value());  // DB write, side effects
    }

    // Only commit after ALL records in this batch are processed
    // If we crash before this line, these records are re-processed on restart
    consumer.commitSync();
}

// Scenario: crash after processOrder() but before commitSync()
// → Kafka re-delivers the same records on restart
// → processOrder() runs again → DUPLICATE

At-most-once commits offsets before processing; a crash after commit but before processing causes data loss — acceptable for metrics but not for financial or order data.

• ✓At-most-once: commit offsets BEFORE processing — crash after commit but before process = data loss, but no duplicates.
• ✓enable.auto.commit=true with a periodic timer creates the at-most-once window automatically.
• ✓Acceptable for: non-critical metrics, telemetry, analytics where occasional loss is negligible.
• ✓Never use for: financial data, order state, inventory, any user-facing mutation.
• ✓Spring Kafka defaults to at-least-once (disables auto-commit); override only intentionally.
• ✓The three semantics compare as: at-most-once (loss OK), at-least-once (duplicate OK), exactly-once (neither, but has overhead).

// At-most-once — commit BEFORE processing (data loss if crash)
Properties props = new Properties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("metrics"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

    // COMMIT FIRST — at-most-once: crash after this but before processing = data loss
    consumer.commitSync();

    for (ConsumerRecord<String, String> record : records) {
        // If we crash here, records are already committed — will NOT be reprocessed
        processMetric(record.value());
    }
}

// enable.auto.commit=true produces the same at-most-once window:
// auto-commit fires after poll() but before processing completes

Achieved by combining idempotent producer, transactions, and isolation.level=read_committed on consumers; Kafka Streams supports EOS with processing.guarantee=exactly_once_v2.

• ✓EOS requires three layers working together: idempotent producer (dedup retries) + transactions (atomic multi-partition) + read_committed consumer.
• ✓Idempotent producer (enable.idempotence=true) prevents duplicate writes caused by retries within a single producer session.
• ✓Transactions allow atomic writes across multiple partitions — either all committed or all aborted, no partial state.
• ✓Consumers must set isolation.level=read_committed to skip aborted transaction records ("dirty reads").
• ✓For Kafka Streams, set processing.guarantee=exactly_once_v2 — Streams manages the entire EOS pipeline automatically.
• ✓EOS adds latency overhead (replication round-trips, transaction coordinator calls) — use it only for financial or critical-correctness use cases.

// Enable idempotent producer (required for EOS)
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Automatically sets:
//   acks=all
//   retries=Integer.MAX_VALUE
//   max.in.flight.requests.per.connection=5

// The broker assigns a PID and tracks sequence numbers per partition
// Duplicate batches (same PID + sequence) are silently discarded by the broker

Exactly-once semantics guarantees each message is delivered and processed precisely once, even during failures. It requires idempotent producers, transactional APIs, and read_committed consumers.

• ✓EOS = idempotent producer + transactional writes + read_committed consumer
• ✓enable.idempotence=true prevents producer-retry duplicates using sequence numbers
• ✓transactional.id must be unique per producer instance across restarts
• ✓read_committed consumers skip messages from aborted transactions
• ✓EOS has ~20% throughput overhead — use only when truly required

@Configuration
public class TransactionalProducerConfig {
    @Bean
    public ProducerFactory<String, Order> producerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        config.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-producer-1");
        config.put(ProducerConfig.ACKS_CONFIG, "all");
        return new DefaultKafkaProducerFactory<>(config);
    }
}

// Atomic publish to multiple topics
kafkaTemplate.executeInTransaction(ops -> {
    ops.send("orders", order.getId(), order);
    ops.send("order-events", order.getId(), new OrderCreatedEvent(order));
    return true; // throw to abort
});

// Consumer must use read_committed
spring.kafka.consumer.properties.isolation.level=read_committed

An idempotent consumer produces the same result whether it processes a message once or multiple times — the practical alternative to EOS for most use cases.

• ✓Idempotent consumer: same result whether message processed once or N times
• ✓Track processed message IDs (topic-partition-offset) in a deduplication table
• ✓Unique DB constraint on business key is a simpler alternative
• ✓Clean up old dedup entries periodically
• ✓Prefer idempotent consumers over EOS — simpler and faster

@Service
@Transactional
public class IdempotentOrderConsumer {
    @Autowired ProcessedMessageRepository repo;
    @Autowired OrderService orderService;

    @KafkaListener(topics = "orders")
    public void handle(ConsumerRecord<String, Order> record) {
        String msgId = record.topic() + "-" + record.partition() + "-" + record.offset();
        if (repo.existsByMessageId(msgId)) return; // already processed
        orderService.processOrder(record.value());
        repo.save(new ProcessedMessage(msgId, Instant.now()));
    }
}