Idempotent Consumer Pattern

Overview

Even with at-least-once delivery, correct behaviour is achievable by making consumers idempotent. Common techniques include a deduplication table keyed by message ID or using database unique constraints.

Deduplication with a Processed Messages Table

@Service
@Transactional
public class IdempotentOrderConsumer {
    @Autowired ProcessedMessageRepository repo;
    @Autowired OrderService orderService;

    @KafkaListener(topics = "orders")
    public void handle(ConsumerRecord<String, Order> record) {
        String msgId = record.topic() + "-" + record.partition() + "-" + record.offset();
        if (repo.existsByMessageId(msgId)) return; // already processed
        orderService.processOrder(record.value());
        repo.save(new ProcessedMessage(msgId, Instant.now()));
    }
}

Key Points to Remember

• 1Idempotent consumer: same result whether message processed once or N times
• 2Track processed message IDs (topic-partition-offset) in a deduplication table
• 3Unique DB constraint on business key is a simpler alternative
• 4Clean up old dedup entries periodically
• 5Prefer idempotent consumers over EOS — simpler and faster