At-Least-Once vs Exactly-Once Delivery
IntermediateWhen a system delivers messages, it makes one of three guarantees. At-most-once may lose messages but never duplicates. At-least-once never loses messages but may deliver duplicates. Exactly-once delivers each message once and only once — the ideal, but genuinely hard in a distributed system with failures and retries. In practice, most systems use at-least-once delivery plus idempotent consumers to achieve effectively-exactly-once results.
Think of confirming a restaurant reservation
At-most-once: you call once and hang up — if the line dropped, you may have no booking, but you will never have two. At-least-once: you keep calling until someone confirms — you definitely have a booking, but you might have accidentally made three. Exactly-once: the restaurant recognises your name and ensures exactly one table no matter how many times you call. That recognition — deduplicating your repeated calls — is how real systems fake exactly-once on top of at-least-once.
Step by Step
Key Concepts
The Three Semantics
At-most-once (may lose, never duplicate), at-least-once (never lose, may duplicate), and exactly-once (never lose, never duplicate). They trade reliability against complexity.
Why Exactly-Once Is Hard
Delivering, processing, and acknowledging must be one atomic step across networks and crashes. Any non-atomic gap lets a failure cause either a lost message or a duplicate.
Idempotent Consumer
A consumer that produces the same result no matter how many times it processes the same message — via dedup by ID or idempotency keys. It turns at-least-once into effectively-exactly-once.
Kafka Transactions
Kafka can atomically consume, transform, and produce within its own system, giving exactly-once stream processing. Effects on external systems still generally require idempotency.
Key Facts
- At-least-once is the default in most messaging systems because losing data is usually worse than occasionally duplicating it.
- True end-to-end exactly-once across arbitrary systems is effectively impossible; the pragmatic answer is at-least-once delivery plus idempotent processing.
- Design consumers to be idempotent from day one — retries and redeliveries are inevitable in any real distributed system.
Real-World Applications
Payment processing
A payment charge must not double-bill on a retry, so the system uses at-least-once delivery with an idempotency key, ensuring a redelivered charge request is recognised and applied only once.
Streaming aggregations
A Kafka Streams job counting events uses Kafka exactly-once transactions so a crash and retry does not double-count, keeping the running totals accurate within the Kafka boundary.
Frequently Asked Questions
What is the difference between at-least-once and exactly-once delivery?
At-least-once guarantees no message is lost, but because the sender retries until it gets an acknowledgement, a message may be delivered more than once (duplicates possible). Exactly-once guarantees each message is delivered and processed once and only once — no loss and no duplicates. Exactly-once is the ideal but is much harder to achieve reliably in a distributed system.
Why is exactly-once delivery so hard?
Because delivering a message, processing it, and recording the acknowledgement must happen atomically, even though networks drop packets and nodes crash. If a node fails after doing the work but before recording it, the message gets reprocessed (a duplicate); if it records before doing the work, a failure can lose it. Closing this gap perfectly across independent systems is effectively impossible, which is why exactly-once is usually approximated.
How do systems achieve exactly-once in practice?
They combine at-least-once delivery with idempotent consumers. The messaging layer ensures nothing is lost (with retries), and the consumer is designed so that processing the same message twice has no extra effect — for example by deduplicating on a message ID or using idempotency keys. This produces effectively-exactly-once results without needing perfect exactly-once delivery.
What is an idempotent consumer?
It is a consumer whose effect is the same no matter how many times it processes a given message. It typically records which message IDs it has already handled, or uses an idempotency key so repeated operations collapse into one. Because messaging systems deliver at-least-once and can redeliver, making consumers idempotent is the standard way to safely tolerate duplicates.