Exactly-Once Semantics (EOS)
AdvancedAchieved by combining idempotent producer, transactions, and isolation.level=read_committed on consumers; Kafka Streams supports EOS with processing.guarantee=exactly_once_v2.
Overview
Exactly-once semantics (EOS) means that even in the presence of producer retries, broker failures, or consumer restarts, each record is processed and its effect is reflected in the output exactly one time — no duplicates, no data loss. Achieving true EOS in a distributed system is hard. Kafka achieves it end-to-end by combining three features: (1) idempotent producers that deduplicate retries per partition, (2) transactional producers that atomically publish to multiple partitions, and (3) consumers with isolation.level=read_committed that skip uncommitted transactional records. Kafka Streams abstracts all of this with processing.guarantee=exactly_once_v2.
Layer 1 — Idempotent Producer
An idempotent producer (enable.idempotence=true) is assigned a PID (Producer ID) by the broker. Every message batch is tagged with the PID and a monotonically increasing sequence number per partition. If the producer retries a failed send, the broker recognises the PID+sequence as a duplicate and discards it — preventing duplicate writes caused by retries.
Idempotent producers provide exactly-once delivery within a single producer session, per partition. They do not span multiple partitions or survive a producer restart (the PID changes on restart). For cross-partition atomicity, transactions are needed.
// Enable idempotent producer (required for EOS)
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Automatically sets:
// acks=all
// retries=Integer.MAX_VALUE
// max.in.flight.requests.per.connection=5
// The broker assigns a PID and tracks sequence numbers per partition
// Duplicate batches (same PID + sequence) are silently discarded by the brokerLayer 2 — Transactional Producer (Atomic Multi-Partition Writes)
Transactions allow a producer to write to multiple partitions atomically. All records in the transaction are either all committed or all aborted — there is no partial state visible to read_committed consumers.
The canonical use case is the consume-transform-produce pattern in stream processing: read from input topic, transform, write results + commit offset atomically. Without transactions, a crash after writing results but before committing the offset causes duplicate processing.
// Transactional producer — atomic multi-partition write
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
try {
producer.beginTransaction();
// Write to two different topics atomically
producer.send(new ProducerRecord<>("order-confirmed", orderId, payload));
producer.send(new ProducerRecord<>("audit-log", orderId, auditPayload));
// Commit offsets + records atomically (consume-transform-produce)
producer.sendOffsetsToTransaction(consumedOffsets, consumerGroupMetadata);
producer.commitTransaction();
} catch (ProducerFencedException | InvalidProducerEpochException e) {
// Another instance has taken over this transactional.id — do not retry
producer.close();
} catch (KafkaException e) {
producer.abortTransaction(); // all records in this tx are discarded
}Layer 3 — Consumer isolation.level & Kafka Streams EOS
Consumers must set isolation.level=read_committed to skip records from aborted transactions and uncommitted transactional writes. Without this setting, consumers see "dirty reads" — records that were later aborted.
For Kafka Streams applications, set processing.guarantee=exactly_once_v2 (EOS-V2, Kafka 2.5+). Streams handles the entire consume-transform-produce cycle with EOS automatically, requiring no manual transaction management.
// Consumer — only read committed transactional records
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
// Kafka Streams — declarative EOS
Properties streamsProps = new Properties();
streamsProps.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-processor");
streamsProps.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
streamsProps.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE_V2); // handles idempotent + transactional internally
StreamsBuilder builder = new StreamsBuilder();
builder.stream("payments")
.filter((k, v) -> isValid(v))
.to("confirmed-payments"); // exactly-once end-to-end
KafkaStreams streams = new KafkaStreams(builder.build(), streamsProps);
streams.start();Key Points to Remember
- 1EOS requires three layers working together: idempotent producer (dedup retries) + transactions (atomic multi-partition) + read_committed consumer.
- 2Idempotent producer (enable.idempotence=true) prevents duplicate writes caused by retries within a single producer session.
- 3Transactions allow atomic writes across multiple partitions — either all committed or all aborted, no partial state.
- 4Consumers must set isolation.level=read_committed to skip aborted transaction records ("dirty reads").
- 5For Kafka Streams, set processing.guarantee=exactly_once_v2 — Streams manages the entire EOS pipeline automatically.
- 6EOS adds latency overhead (replication round-trips, transaction coordinator calls) — use it only for financial or critical-correctness use cases.
Interview Questions
Sign in to ask AriaWhat does exactly-once semantics mean in Kafka and how is it achieved?
What is the difference between idempotent producer and transactional producer in Kafka?
Why do consumers need isolation.level=read_committed for EOS?
What is the consume-transform-produce pattern and how do Kafka transactions make it exactly-once?
When would you NOT use exactly-once semantics in a Kafka application?
Ask Aria about Exactly-Once Semantics (EOS)
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.