Error Handling Patterns
IntermediateCombine retry (with back-off), DLX, and poison-message detection to build robust error handling; alert on DLQ growth and build replay tooling for production incidents.
Overview
Robust RabbitMQ error handling combines three complementary mechanisms. First, in-process retry with exponential back-off for transient failures (database hiccup, downstream timeout) — these are retried immediately in memory without re-queuing. Second, the Dead Letter Exchange (DLX) for messages that exhaust retries or are explicitly rejected — they land in a Dead Letter Queue (DLQ) for investigation and manual replay. Third, poison-message detection to identify messages that crash the consumer on every attempt — these must be discarded or quarantined rather than retried indefinitely. Spring AMQP's RetryInterceptorBuilder and DeadLetterPublishingRecoverer implement this pattern declaratively, with full observability into what failed and why.
In-process retry with exponential back-off
Spring AMQP's MessageRecoverer is invoked after all retries are exhausted. RetryInterceptorBuilder wires the retry policy to the listener container.
@Configuration
public class ErrorHandlingConfig {
@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
ConnectionFactory connectionFactory,
RabbitTemplate rabbitTemplate) {
SimpleRabbitListenerContainerFactory factory =
new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setAcknowledgeMode(AcknowledgeMode.AUTO); // nack triggers retry
// After 3 retries → dead-letter the message
factory.setAdviceChain(
RetryInterceptorBuilder.stateless()
.maxAttempts(3)
.backOffOptions(500, 2.0, 8000) // 0.5s, 1s, 2s → DLQ
.recoverer(new RejectAndDontRequeueRecoverer()) // nack + no requeue
.build()
);
return factory;
}
}
@RabbitListener(queues = "orders.queue")
public void processOrder(OrderEvent event) {
// If this throws, Spring AMQP retries up to 3x before dead-lettering
orderService.process(event);
}DLX setup and DLQ routing
Configure the working queue to dead-letter to a specific exchange on rejection. A DLQ bound to the DLX receives all failed messages with x-death metadata for diagnosis.
@Configuration
public class QueueConfig {
// Working queue — failed messages routed to DLX
@Bean
public Queue ordersQueue() {
return QueueBuilder.durable("orders.queue")
.withArgument("x-dead-letter-exchange", "orders.dlx")
.withArgument("x-dead-letter-routing-key", "orders.dead")
.withArgument("x-message-ttl", 3600000) // 1h max live time
.build();
}
// Dead letter exchange
@Bean
public DirectExchange deadLetterExchange() {
return new DirectExchange("orders.dlx");
}
// Dead letter queue — messages here need investigation
@Bean
public Queue deadLetterQueue() {
return QueueBuilder.durable("orders.dlq").build();
}
@Bean
public Binding dlqBinding() {
return BindingBuilder.bind(deadLetterQueue())
.to(deadLetterExchange())
.with("orders.dead");
}
}
// Dead-lettered messages carry x-death headers:
// x-death[0].count — how many times dead-lettered
// x-death[0].reason — "rejected" | "expired" | "maxlen"
// x-death[0].queue — originating queue
// x-death[0].time — timestamp of deathPoison message detection and replay tooling
A poison message is one that crashes the consumer on every attempt. Detect by checking x-death count; quarantine it. Replay by re-publishing from DLQ after the root cause is fixed.
// Poison message detector — quarantine if dead-lettered >5 times
@RabbitListener(queues = "orders.dlq")
public void handleDeadLetter(
Message message,
@Header(value = "x-death", required = false) List<Map<String, Object>> xDeath,
Acknowledgment ack) {
int deathCount = xDeath != null ? xDeath.stream()
.mapToInt(d -> ((Long) d.get("count")).intValue()).sum() : 0;
if (deathCount > 5) {
// Poison message — log for human review, ack to remove from DLQ
log.error("POISON MESSAGE detected after {} deaths: {}",
deathCount, new String(message.getBody()));
alertService.notifyPoisonMessage(message);
ack.acknowledge(); // remove from DLQ
} else {
// Replay after a delay
rabbitTemplate.send("orders.exchange", "new-order", message);
ack.acknowledge();
}
}
// CLI replay: re-publish all DLQ messages during off-peak
// (use RabbitMQ Shovel plugin or shovel via Management HTTP API)
// POST /api/shovels/vhost {"name":"dlq-replay","src-queue":"orders.dlq",
// "dest-exchange":"orders.exchange", "dest-routing-key":"new-order"}Key Points to Remember
- 1Retry transient failures in-process (RetryInterceptorBuilder); dead-letter permanent failures for investigation.
- 2RejectAndDontRequeueRecoverer sends nack + no-requeue after exhausting retries, triggering the DLX routing.
- 3x-death headers on DLQ messages record the death count, reason (rejected/expired/maxlen), and originating queue.
- 4Alert on DLQ depth growing — it means messages are failing consistently; investigate before the DLQ fills.
- 5Poison messages loop forever without a death-count check; quarantine messages that die more than N times.
- 6Build replay tooling before an incident — being able to re-publish DLQ messages after a bug fix is critical for recovery.
Interview Questions
Sign in to ask AriaWhat is the difference between nack with requeue=true and nack with requeue=false?
How would you build a retry-with-backoff mechanism using RabbitMQ without Spring AMQP?
Explain the x-death header and how it helps you detect poison messages.
What happens to a message that is dead-lettered when the DLQ is also full?
How would you implement a replay mechanism to re-process all failed messages after fixing a bug?
Ask Aria about Error Handling Patterns
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.