Message Queues

Intermediate

Communication Patterns

Message queues (RabbitMQ, SQS, Kafka) decouple producers and consumers, enabling asynchronous processing, load levelling, and fault-tolerant communication between services.

Overview

A message queue is middleware that accepts messages from producers and delivers them to consumers asynchronously. The queue acts as a buffer — producers can publish at their own rate without waiting for consumers to process. This decouples services temporally (producer and consumer do not need to be available at the same time) and in terms of throughput (the queue absorbs traffic spikes). Common use cases include order processing, email/notification sending, log aggregation, and task scheduling. Key features include guaranteed delivery (at-least-once), ordering (FIFO queues), dead-letter queues (DLQ) for failed messages, and visibility timeouts. Major implementations include RabbitMQ (AMQP, push-based), Amazon SQS (managed, pull-based), and Apache Kafka (distributed log, high throughput).

Core Concepts

Producers send messages to a queue. Consumers poll or receive messages from the queue. After successful processing, the consumer acknowledges the message and the queue removes it.

Conceptual + Java — SQS message queue

// Message queue flow
//
// Producer → [Queue] → Consumer
//
// 1. Producer sends message to queue
// 2. Queue persists message durably
// 3. Consumer pulls (or queue pushes) message
// 4. Consumer processes message
// 5. Consumer ACKs → queue deletes message
// 6. If no ACK within timeout → message redelivered (at-least-once)

// Amazon SQS example (Java SDK v2)
// Send
sqsClient.sendMessage(SendMessageRequest.builder()
    .queueUrl(queueUrl)
    .messageBody("{"orderId":"123","action":"process"}")
    .delaySeconds(0)
    .build());

// Receive + process + delete
List<Message> messages = sqsClient.receiveMessage(r -> r
    .queueUrl(queueUrl)
    .maxNumberOfMessages(10)
    .waitTimeSeconds(20)  // long polling
).messages();

for (Message msg : messages) {
    processOrder(msg.body());
    sqsClient.deleteMessage(r -> r.queueUrl(queueUrl).receiptHandle(msg.receiptHandle()));
}

Dead-Letter Queues & Retry

Messages that fail processing repeatedly are moved to a dead-letter queue (DLQ) for investigation. This prevents poison messages from blocking the main queue.

Config + Java — dead-letter queue setup

// Dead-letter queue flow
//
// Main Queue → Consumer fails → retry 3x → Dead-Letter Queue
//
// SQS redrive policy
{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123:orders-dlq",
  "maxReceiveCount": 3
}
// After 3 failed attempts, message moves to DLQ

// Spring Boot with RabbitMQ DLQ
@Bean
public Queue mainQueue() {
    return QueueBuilder.durable("orders")
        .withArgument("x-dead-letter-exchange", "")
        .withArgument("x-dead-letter-routing-key", "orders-dlq")
        .withArgument("x-message-ttl", 60000)  // 60s before retry
        .build();
}

@Bean
public Queue deadLetterQueue() {
    return QueueBuilder.durable("orders-dlq").build();
}

// DLQ monitoring: alert when DLQ depth > threshold
// Periodically inspect + replay or discard DLQ messages

Queue vs Log (SQS vs Kafka)

Traditional queues (SQS, RabbitMQ) delete messages after consumption — each message is processed once. Kafka retains messages as a log — multiple consumers can read the same message, and consumers can replay from any offset.

Conceptual — queue vs event log comparison

// Traditional queue (SQS / RabbitMQ)
// Message consumed → deleted → gone
// One consumer group processes each message once
// Good for: task queues, job processing, point-to-point

// Event log (Kafka)
// Message consumed → retained for days/weeks
// Multiple consumer groups read independently
// Consumers can replay from any offset
// Good for: event sourcing, CDC, fan-out to many services

// Comparison
// Feature          | SQS/RabbitMQ    | Kafka
// ─────────────────────────────────────────────
// Message lifecycle| Delete after ACK| Retained (TTL)
// Consumer model   | Competing       | Consumer groups
// Replay           | ❌ No           | ✅ Yes
// Ordering         | FIFO optional   | Per-partition
// Throughput       | Moderate        | Very high
// Complexity       | Simple          | Higher

Key Points to Remember

1Message queues decouple producers and consumers — enabling async processing and load levelling.
2At-least-once delivery means consumers must be idempotent (handle duplicate messages).
3Dead-letter queues catch messages that fail processing repeatedly — essential for production systems.
4Traditional queues (SQS, RabbitMQ) delete after consumption; Kafka retains for replay.
5Use queues for task distribution; use Kafka for event streaming and fan-out to multiple consumers.

Interview Questions

What are the benefits of using a message queue between services?

EasyTCS

What is a dead-letter queue and why is it important?

EasyInfosys

How do you ensure messages are processed exactly once?

MediumAmazon

Compare SQS, RabbitMQ, and Kafka — when would you use each?

MediumGoogle

Design an order processing pipeline that handles 100K orders/minute with retries and DLQ.

HardFlipkart

Ask Aria about Message Queues

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

REST vs gRPC

Event-Driven Architecture