Kafka Design Patterns
AdvancedEvent notification, event-carried state transfer, event sourcing, CQRS with Kafka, the outbox pattern, and the inbox pattern are common Kafka-based architectural patterns.
Overview
Kafka's append-only, replayable log enables a rich set of architectural patterns beyond basic messaging. **Event Notification**: publish an event when something happens; consumers decide whether to act (thin events with just an ID). **Event-Carried State Transfer**: embed the full state in the event so consumers never need to call back — avoids coupling. **Outbox Pattern**: write events to a DB outbox table atomically with state changes, then a separate poller publishes to Kafka — eliminates the dual-write problem. **Inbox Pattern**: persist received events to a local inbox table for idempotent, ordered processing. **CQRS with Kafka**: commands update the write model; events on Kafka update read projections. Choosing the right pattern prevents the most common Kafka anti-patterns.
Event Notification vs Event-Carried State Transfer
**Event Notification** emits thin events (just entity ID + event type). Consumers call back to the source service to get full details. Simple but creates coupling on the read path. **Event-Carried State Transfer** embeds the full state in the event body. Consumers are fully autonomous — no callback needed. Preferred for high-read-rate downstream projections or cross-region replication.
// Event Notification — thin event (ID only)
public record OrderPlacedEvent(String orderId, Instant occurredAt) {}
// Consumer receives orderId, calls back: GET /orders/{orderId}
// ❌ Coupling: consumer fails if order-service is down
// Event-Carried State Transfer — full state in event
public record OrderPlacedEvent(
String orderId,
String customerId,
List<OrderLine> lines,
BigDecimal total,
String currency,
Address shippingAddress,
Instant occurredAt
) {}
// Consumer is fully autonomous — no callback needed
// ✓ Resilient: works even if order-service is temporarily down
// ✓ Replayable: projections rebuilt from past events include all data
// When to prefer notification:
// - Event payload would be huge and rarely needed
// - Data is sensitive (PII) and should not be in the event log
// - Low-volume events where a callback is acceptableOutbox Pattern — Solving the Dual-Write Problem
The dual-write problem: you update the DB and then publish to Kafka — but if the app crashes between the two, the DB is updated and the event is lost (or vice versa). The **Outbox Pattern** uses a single DB transaction to write both the business state change AND an `outbox_events` row. A separate **CDC relay** (Debezium) or poller reads the outbox table and publishes to Kafka, guaranteeing at-least-once delivery with transactional consistency.
// Step 1: Write state + outbox event in ONE transaction
@Transactional
public Order placeOrder(CreateOrderRequest req) {
Order order = orderRepo.save(new Order(req));
// Atomically record the event to publish
outboxRepo.save(OutboxEvent.builder()
.aggregateId(order.getId().toString())
.eventType("order.placed")
.payload(objectMapper.writeValueAsString(new OrderPlacedEvent(order)))
.build());
return order; // event published ONLY after this commit
}
// Step 2: Outbox relay — poll and publish (or use Debezium CDC)
@Scheduled(fixedDelay = 500)
@Transactional
void publishOutbox() {
List<OutboxEvent> pending = outboxRepo.findTop100ByPublishedFalseOrderById();
pending.forEach(e -> {
kafkaTemplate.send(e.getEventType(), e.getAggregateId(), e.getPayload());
e.setPublished(true);
});
outboxRepo.saveAll(pending);
}
// Better: Debezium CDC captures outbox table inserts → zero polling latency
// table: outbox_events → Kafka topic: order.placedInbox Pattern and CQRS with Kafka
The **Inbox Pattern** is the consumer-side complement to the Outbox: persist each received event to a local `inbox_events` table (keyed by event ID) before processing. This makes processing idempotent (duplicate events are detected) and allows ordered retry. **CQRS with Kafka**: commands go to the write model (DB); domain events published to Kafka update multiple independent read projections — each tailored to a specific query's access pattern.
// Inbox Pattern — idempotent consumer
@KafkaListener(topics = "order-placed")
@Transactional
public void handle(OrderPlacedEvent event) {
// Guard: skip if already processed (idempotency)
if (inboxRepo.existsByEventId(event.eventId())) {
return; // duplicate — acknowledge and skip
}
inboxRepo.save(new InboxEvent(event.eventId(), "processing"));
try {
fulfilmentService.startFulfilment(event);
inboxRepo.updateStatus(event.eventId(), "processed");
} catch (Exception e) {
inboxRepo.updateStatus(event.eventId(), "failed");
throw e; // NACK — retry via @RetryableTopic
}
}
// CQRS with Kafka — multiple projections from one event stream
// Command: POST /orders → write model (orders DB)
// Event: OrderPlacedEvent on Kafka
// → Projection 1: order-summary (optimised for list views)
// → Projection 2: analytics-fact-table (optimised for reports)
// → Projection 3: notification-queue (send confirmation email)
// → Projection 4: inventory-reservation (check + reserve stock)Key Points to Remember
- 1Event Notification: thin event with ID; consumers call back — simple but coupled
- 2Event-Carried State Transfer: full state in event; consumers autonomous — preferred for projections
- 3Outbox Pattern: write state + event in one DB transaction; relay publishes to Kafka — no dual-write
- 4Inbox Pattern: store received events in local table for idempotent, ordered processing
- 5CQRS: commands update write model; events update multiple independent read projections
- 6Debezium CDC is the most reliable Outbox relay — zero polling latency, no app-level poller needed
Interview Questions
Sign in to ask AriaWhat is the dual-write problem and how does the Outbox Pattern solve it?
When would you choose event-carried state transfer over event notification?
What is the Inbox Pattern and why does it make consumers idempotent?
How does CQRS with Kafka allow multiple read models with different access patterns?
What is the role of Debezium in the Outbox Pattern?
Ask Aria about Kafka Design Patterns
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.