Home/Learn/Apache Kafka/Idempotent Producer

Idempotent Producer

Intermediate

Producers

enable.idempotence=true assigns a producer ID and sequence number to each record; the broker deduplicates retries, guaranteeing exactly-once delivery per partition.

Overview

By default, Kafka's "at-least-once" producer may produce duplicates when retrying after a network error: the broker may have already committed the message but the ack was lost. The **idempotent producer** (`enable.idempotence=true`) eliminates this by assigning each producer instance a unique **Producer ID (PID)** and tagging every record batch with a monotonically increasing **sequence number** per partition. If the broker receives a batch with a sequence number it has already seen for that PID+partition combination, it discards the duplicate and returns success to the producer. This gives exactly-once semantics per partition within a single producer session — without requiring Kafka Transactions. It also forces `acks=all`, `retries=MAX_INT`, and `max.in.flight.requests.per.connection≤5`.

Enabling and Verifying Idempotence

Set `enable.idempotence=true` on the producer. Spring Kafka's `KafkaTemplate` delegates to the underlying producer — set the property in `application.properties` or the producer factory. The implicit settings changes (`acks`, `retries`, `max.in.flight`) are applied automatically; setting them to incompatible values throws a `ConfigException` at startup.

Kafka — enable.idempotence config

# application.properties (Spring Kafka)
spring.kafka.producer.properties.enable.idempotence=true
# Implicitly sets:
#   acks=all
#   retries=Integer.MAX_VALUE
#   max.in.flight.requests.per.connection=5

# Java producer config
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// The following are set automatically but can be overridden compatibly:
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

How the Broker Deduplicates

The broker maintains a per-(PID, partition) window of the last 5 accepted sequence numbers. On receiving a batch, it checks: if `seq == last_seq + 1` → accept; if `seq <= last_seq` → duplicate, discard and ack; if `seq > last_seq + 1` → out-of-order, return error. The PID is ephemeral — if the producer restarts, it gets a new PID and the idempotency guarantee resets (transactional producers solve this).

Kafka — broker deduplication logic

// Broker deduplication logic (conceptual):
// Per (PID, Partition): maintain last committed sequence number

received_seq = 10  (retry of already-committed batch)
last_committed_seq = 10

if received_seq == last_committed_seq + 1:
    commit and ack         // new record
elif received_seq <= last_committed_seq:
    discard and ack        // duplicate — idempotency in action
else:
    return OutOfOrderSequenceException  // gap in sequence

// What happens on producer restart?
// New PID is assigned → sequence resets to 0
// Old PID window is eventually expired from broker memory
// → idempotency is per-session only (use transactions for cross-session EOS)

Idempotent vs Transactional Producer

The idempotent producer gives exactly-once semantics for a single producer writing to multiple partitions **within a session**. It does NOT guarantee atomicity across multiple topic-partitions (e.g., read-process-write pipelines). For that, use the **transactional producer**, which builds on top of idempotence to provide atomic multi-partition writes and consume-transform-produce (EOS) pipelines.

Kafka — idempotent vs transactional producer

// Idempotent: exactly-once per partition, within session
producer.send(new ProducerRecord<>("topic-a", key, value));
// If retry occurs, broker deduplicates → no duplicate

// Transactional: atomically write to MULTIPLE partitions
producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders", orderId, orderJson));
    producer.send(new ProducerRecord<>("inventory", itemId, reserveJson));
    producer.commitTransaction();   // both writes visible atomically
} catch (ProducerFencedException e) {
    producer.close();               // another instance took over (fenced)
} catch (KafkaException e) {
    producer.abortTransaction();    // roll back both writes
}

Key Points to Remember

1enable.idempotence=true assigns each producer a PID and adds per-partition sequence numbers
2Broker deduplicates retried batches by checking the PID+partition sequence window
3Idempotence forces acks=all, retries=MAX_INT, max.in.flight≤5 — set automatically
4PID is ephemeral: producer restart gets a new PID, resetting the deduplication window
5Idempotent producer gives exactly-once per partition per session — not cross-session
6Transactional producer builds on idempotence to add atomic multi-partition writes

Interview Questions

What does enable.idempotence=true do under the hood in Kafka?

MediumConfluent

Why does idempotent producer force max.in.flight.requests.per.connection≤5?

HardLinkedIn

What happens to idempotency when the producer restarts and gets a new PID?

HardUber

What is the difference between an idempotent and a transactional producer?

MediumAmazon

Can the idempotent producer guarantee no duplicates if the consumer also retries?

HardZalando

Ask Aria about Idempotent Producer

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Message Keys & Partitioning Strategy

Transactional Producer