Kafka Messaging

Advanced

Production & Advanced

Spring Kafka wraps the Kafka client with KafkaTemplate for producing and @KafkaListener for consuming. Understanding partitions, consumer groups, and error handling is essential for production event-driven architectures.

Overview

Kafka is a distributed log — producers append messages to partitions, consumers read from offsets. Consumer groups enable parallel consumption: each partition is assigned to exactly one consumer in a group, so adding consumers scales throughput linearly up to the partition count. Spring Kafka's @KafkaListener handles deserialization, offset commits, and error handling. KafkaTemplate.send() returns a CompletableFuture — always handle failures to prevent silent message loss.

KafkaTemplate — Producing Messages

KafkaTemplate is the producer API. Send messages to a topic with an optional key — same key always goes to the same partition (ordering guarantee). Always handle send failures to avoid silent data loss.

YAML + Java — KafkaTemplate with async send and error handling

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

# application.yml
spring:
  kafka:
    bootstrap-servers: ${KAFKA_BROKERS:localhost:9092}
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all             # wait for all replicas before acknowledging
      retries: 3
      properties:
        enable.idempotence: true  # exactly-once producer semantics

@Service
public class OrderEventProducer {

    private static final String TOPIC = "order-events";
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    @Autowired
    public OrderEventProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrderPlaced(Order order) {
        OrderEvent event = new OrderEvent(order.getId(), "ORDER_PLACED",
                                          order.getCustomerId(), Instant.now());

        // Key = customerId → same customer's orders go to same partition (ordered)
        kafkaTemplate.send(TOPIC, order.getCustomerId(), event)
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    log.error("Failed to publish order event: {}", order.getId(), ex);
                    // Dead-letter, alert, or store for retry
                } else {
                    log.info("Published to partition {} offset {}",
                        result.getRecordMetadata().partition(),
                        result.getRecordMetadata().offset());
                }
            });
    }
}

@KafkaListener — Consuming Messages

Consumer groups let multiple instances of your service share the message load. Error handling with DeadLetterPublishingRecoverer sends failed messages to a DLT (Dead Letter Topic) after N retries instead of blocking the consumer.

YAML + Java — @KafkaListener with concurrency and dead-letter handling

# application.yml
spring:
  kafka:
    consumer:
      group-id: order-processor
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.aicancode.*"

@Component
public class OrderEventConsumer {

    @KafkaListener(topics = "order-events", groupId = "order-processor",
                   concurrency = "3")  // 3 consumer threads per instance
    public void handleOrderEvent(OrderEvent event,
                                  @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
                                  @Header(KafkaHeaders.OFFSET) long offset) {
        log.info("Processing event {} from partition {} offset {}",
                 event.orderId(), partition, offset);
        inventoryService.reserve(event);
        // If this throws RuntimeException → retried, then sent to DLT
    }
}

// Error handling — retry 3 times, then send to {topic}.DLT
@Configuration
public class KafkaErrorConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(template); // sends to order-events.DLT

        ExponentialBackOffWithMaxRetries backOff =
            new ExponentialBackOffWithMaxRetries(3);
        backOff.setInitialInterval(1000L);  // 1s, 2s, 4s then DLT
        backOff.setMultiplier(2.0);

        return new DefaultErrorHandler(recoverer, backOff);
    }
}

Key Points to Remember

1Producer key determines partition assignment — same key always goes to the same partition (ordering guarantee).
2acks=all + enable.idempotence=true gives at-least-once delivery with duplicate protection on the producer side.
3Consumer group ID determines the group — each partition is assigned to one consumer per group for parallel processing.
4concurrency on @KafkaListener sets the number of consumer threads — max useful value equals the partition count.
5DeadLetterPublishingRecoverer sends failed messages to {topic}.DLT after exhausting retries — prevents consumer blocking.
6Auto-commit is dangerous — use manual offset commit (AckMode.MANUAL) for exactly-once processing guarantees.

Interview Questions

How does a Kafka consumer group enable parallel message processing?

MediumAmazon

What is the role of the message key in Kafka partitioning?

EasyRazorpay

What is a Dead Letter Topic and when would a message be sent there?

MediumThoughtWorks

What is the maximum parallelism for a Kafka consumer group with 6 partitions?

MediumGoldman Sachs

How do you achieve exactly-once semantics in a Kafka consumer?

HardNetflix

Ask Aria about Kafka Messaging

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Application Events

Feign Client