Home/Learn/Apache Kafka/Dead Letter Topic (DLT)

Dead Letter Topic (DLT)

Intermediate

Spring Integration

Failed records after exhausting retries are forwarded to a DLT (topic-name.DLT); a separate listener monitors the DLT for alerting, inspection, or replaying.

Overview

A Dead Letter Topic (DLT) is a Kafka topic that receives messages which could not be successfully processed after all retry attempts are exhausted. Unlike a RabbitMQ DLQ (where the broker routes messages), a Kafka DLT is application-managed: the consumer application forwards the failed record to the DLT topic. Spring Kafka's @RetryableTopic handles this automatically — it creates retry topics with back-off delays and a final DLT topic, forwarding records through the chain. The DLT gives operators a place to inspect, alert on, re-process, or discard permanently failed messages. Monitoring DLT depth is an important SLA signal.

DLT with @RetryableTopic — Automatic Setup

@RetryableTopic creates retry topics and a DLT automatically. Naming convention: `{topic}-retry-0`, `{topic}-retry-1`, …, `{topic}.DLT`. Records flow through retry topics with configurable back-off between attempts. After the last retry, the record is forwarded to the DLT. The @DltHandler method handles DLT records in the same class.

Java — @RetryableTopic + @DltHandler

@Component
@RequiredArgsConstructor
public class PaymentConsumer {

    private final PaymentService paymentService;
    private final AlertService alertService;

    // Spring Kafka creates automatically:
    //   payments-retry-0  (1s delay)
    //   payments-retry-1  (2s delay)
    //   payments-retry-2  (4s delay)
    //   payments.DLT      (after 3 retries exhausted)
    @RetryableTopic(
        attempts = "4",    // 1 original + 3 retries
        backoff  = @Backoff(delay = 1000, multiplier = 2.0),
        dltTopicSuffix = ".DLT",
        autoCreateTopics = "true",
        kafkaTemplate = "kafkaTemplate"
    )
    @KafkaListener(topics = "payments", groupId = "payment-service")
    public void onPayment(ConsumerRecord<String, Payment> record) {
        paymentService.process(record.value());
        // Throws RuntimeException → forwarded to retry topic
        // After 3 retries → forwarded to payments.DLT
    }

    // DLT handler — called when record lands in payments.DLT
    @DltHandler
    public void handleDlt(
            ConsumerRecord<String, Payment> record,
            @Header(KafkaHeaders.RECEIVED_TOPIC)  String topic,
            @Header(KafkaHeaders.EXCEPTION_MESSAGE) String errorMsg) {

        log.error("Payment permanently failed after retries. topic={} key={} error={}",
                  topic, record.key(), errorMsg);

        // Options:
        // 1. Alert the on-call team
        alertService.critical("Payment DLT: " + record.key(), errorMsg);
        // 2. Persist to a failed_payments table for manual inspection
        // 3. Emit a compensating event
    }
}

Manual DLT Publishing — Custom Error Handler

Without @RetryableTopic, configure a DefaultErrorHandler with a DeadLetterPublishingRecoverer on the listener container. This gives you full control over which exceptions trigger DLT routing and what headers to attach.

Java — DeadLetterPublishingRecoverer

@Configuration
@RequiredArgsConstructor
public class KafkaConsumerConfig {

    private final KafkaTemplate<Object, Object> kafkaTemplate;

    @Bean
    public DefaultErrorHandler errorHandler() {
        // After exhausting retries, publish to {topic}.DLT
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(kafkaTemplate,
                // Custom DLT topic: original-topic.DLT
                (record, ex) -> new TopicPartition(
                    record.topic() + ".DLT",
                    record.partition()  // maintain partition affinity
                )
            );

        // Back-off: retry 3 times with 1s, 2s, 4s intervals
        FixedBackOff backOff = new FixedBackOff(1_000L, 3L);
        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

        // Don't retry on permanent errors — go straight to DLT
        handler.addNotRetryableExceptions(
            SchemaValidationException.class,
            InvalidPaymentException.class
        );

        return handler;
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Object> factory(
            ConsumerFactory<String, Object> cf) {
        var factory = new ConcurrentKafkaListenerContainerFactory<String, Object>();
        factory.setConsumerFactory(cf);
        factory.setCommonErrorHandler(errorHandler());
        return factory;
    }
}

DLT Monitoring and Replay

The DLT should be treated as an ops workflow, not a graveyard. Monitor DLT depth as an SLA metric; alert when records arrive. For replay, consume the DLT and re-publish records to the original topic after fixing the underlying issue.

Java — DLT Monitoring + Replay

// DLT monitoring listener — alerts on any DLT message
@Component
public class DltMonitor {

    @KafkaListener(topicPattern = ".*\.DLT", groupId = "dlt-monitor")
    public void onDltRecord(
            ConsumerRecord<String, String> record,
            @Header(KafkaHeaders.EXCEPTION_MESSAGE) String error,
            @Header(KafkaHeaders.EXCEPTION_CAUSE_FQCN) String causeClass) {

        Metrics.counter("kafka.dlt.messages",
            "topic", record.topic(), "cause", causeClass).increment();

        alertService.warn(String.format(
            "DLT message: topic=%s key=%s error=%s", record.topic(), record.key(), error));
    }
}

// Replay — re-publish DLT records to original topic after fix
@Service
@RequiredArgsConstructor
public class DltReplayService {
    private final KafkaTemplate<String, String> kafka;

    public void replay(String dltTopic, int maxMessages) {
        String originalTopic = dltTopic.replace(".DLT", "");
        // Consume from DLT at-most maxMessages, republish to original topic
        // Typically a one-off admin operation via Actuator endpoint or CLI tool
    }
}

Key Points to Remember

1A DLT receives records that could not be processed after all retries are exhausted — application-managed in Kafka (unlike broker-managed in RabbitMQ).
2@RetryableTopic creates retry topics and a DLT automatically; @DltHandler handles DLT records in the same class.
3DeadLetterPublishingRecoverer with DefaultErrorHandler provides manual DLT routing with exception-specific configuration.
4addNotRetryableExceptions() routes records immediately to the DLT for permanent errors (schema validation, invalid data) without wasting retries.
5Monitor DLT depth as an SLA metric — alert on any DLT arrival; a growing DLT indicates a systemic processing failure.
6Replay DLT records after fixing the root cause by re-publishing to the original topic — keep the DLT consumer idempotent.

Interview Questions

What is a Kafka Dead Letter Topic and how does it differ from a RabbitMQ DLQ?

MediumAmazon

How does @RetryableTopic implement retry and DLT routing automatically?

MediumUber

How do you handle both transient and permanent errors differently in a Kafka consumer?

HardNetflix

How would you replay messages from a DLT after fixing a processing bug?

MediumFlipkart

What metrics would you monitor on a DLT to detect production incidents early?

MediumGoogle

Ask Aria about Dead Letter Topic (DLT)

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

@KafkaListener

Message Compression