Idempotency

Intermediate

Reliability & Fault Tolerance

An operation is idempotent if performing it multiple times produces the same result as performing it once. Idempotency is essential in distributed systems where retries and duplicate messages are inevitable.

Overview

In distributed systems, messages can be delivered more than once — network retries, consumer re-processing after a crash, or duplicate API calls from a flaky client. Without idempotency, duplicates cause incorrect behaviour: double charges, duplicate orders, inflated counters. GET, PUT, and DELETE are naturally idempotent in HTTP. POST is not — creating the same resource twice creates two copies. To make non-idempotent operations safe, use idempotency keys: the client sends a unique key with each request, and the server checks if it has already processed that key. If yes, it returns the cached response. Idempotency keys are standard in payment APIs (Stripe, Razorpay) and are equally important for internal message consumers.

Why Idempotency Matters

Without idempotency, retries cause duplicate side effects. With idempotency, the same request processed N times has the same effect as processing it once.

Conceptual — idempotency key prevents double charge

// Without idempotency — double charge!
// Client → POST /payments { amount: 100 }  → 201 (charged)
// Network timeout — client retries
// Client → POST /payments { amount: 100 }  → 201 (charged AGAIN!)
// User charged $200 instead of $100

// With idempotency key — safe retry
// Client → POST /payments
//   Headers: Idempotency-Key: "pay-abc-123"
//   Body: { amount: 100 }
//   → 201 (charged, response cached with key)
//
// Client retries (same key)
// Client → POST /payments
//   Headers: Idempotency-Key: "pay-abc-123"
//   → 200 (return cached response, no double charge)

// HTTP methods and idempotency:
// GET    → idempotent (read-only)
// PUT    → idempotent (full replace)
// DELETE → idempotent (delete same resource = still gone)
// POST   → NOT idempotent (needs idempotency key)
// PATCH  → depends on implementation

Server-Side Implementation

Store idempotency keys with their responses. On duplicate requests, return the stored response. Use a TTL to expire old keys. Handle concurrent duplicate requests with locking.

Java — idempotency key implementation

// Idempotency key implementation (Spring Boot)
@PostMapping("/api/v1/payments")
public ResponseEntity<PaymentResult> charge(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody PaymentRequest req) {

    // 1. Check if already processed
    Optional<CachedResponse> cached = idempotencyStore.get(idempotencyKey);
    if (cached.isPresent()) {
        return ResponseEntity.status(cached.get().status())
            .body(cached.get().body());
    }

    // 2. Acquire lock (prevent concurrent duplicates)
    if (!idempotencyStore.tryLock(idempotencyKey)) {
        return ResponseEntity.status(409).body(PaymentResult.conflict());
    }

    try {
        // 3. Process payment
        PaymentResult result = paymentGateway.charge(req);

        // 4. Cache response with TTL
        idempotencyStore.save(idempotencyKey, result, Duration.ofHours(24));
        return ResponseEntity.status(201).body(result);
    } finally {
        idempotencyStore.unlock(idempotencyKey);
    }
}

// Redis-based idempotency store
// SET idempotency:pay-abc-123 '{"status":201,...}' EX 86400 NX

Idempotent Consumers

Message queue consumers must also be idempotent. Track processed message/event IDs in a database. Use database UPSERT or unique constraints to prevent duplicate processing.

Java — idempotent Kafka consumer

// Idempotent Kafka consumer
@KafkaListener(topics = "order-events")
public void onOrderEvent(OrderEvent event) {
    // Deduplicate using unique event ID
    boolean inserted = jdbcTemplate.update(
        "INSERT INTO processed_events (event_id, processed_at) VALUES (?, NOW()) " +
        "ON CONFLICT (event_id) DO NOTHING",
        event.getEventId()
    ) > 0;

    if (!inserted) {
        log.info("Duplicate event skipped: {}", event.getEventId());
        return;
    }

    // Process event (only runs once per event ID)
    processOrder(event);
}

// Alternative: use natural idempotency
// Instead of: UPDATE balance SET amount = amount + 100
// Use:        UPDATE balance SET amount = 1100 WHERE version = 5
//             (same result no matter how many times executed)

Key Points to Remember

1Idempotency ensures performing an operation N times has the same effect as performing it once.
2Idempotency keys prevent duplicate side effects for non-idempotent operations (POST).
3Store idempotency keys with cached responses; return cached response on duplicates.
4Message consumers must be idempotent — track processed event IDs with unique constraints.
5Use absolute values instead of deltas when possible (SET balance = 1100 vs balance += 100).

Interview Questions

What is idempotency and why is it important in distributed systems?

EasyInfosys

Which HTTP methods are naturally idempotent?

EasyTCS

How would you implement an idempotency key for a payment API?

MediumAmazon

How do you make a Kafka consumer idempotent?

MediumFlipkart

Design a distributed order creation system that guarantees exactly-once processing.

HardGoogle

Ask Aria about Idempotency

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Retry & Exponential Backoff

Rate Limiting