Home/Learn/RabbitMQ/Competing Consumers Pattern

Competing Consumers Pattern

Intermediate

Messaging Patterns

Multiple consumer instances all subscribe to the same queue; the broker delivers each message to exactly one consumer, providing natural horizontal scaling for processing.

Overview

The Competing Consumers pattern (also called Worker Queue or Task Queue) is the primary horizontal scaling mechanism in RabbitMQ. All consumer instances bind to the same queue; RabbitMQ delivers each message to exactly one consumer using round-robin (by default). Adding more consumers linearly increases throughput. Combine with `prefetch=1` and manual acknowledgements for fair dispatch: a consumer only receives a new message once it has acknowledged the previous one, preventing fast consumers from stealing work while slow consumers are backed up. This pattern is ideal for CPU-bound or I/O-bound tasks (image processing, email sending, order fulfilment) that can be parallelised.

Basic Setup with Multiple Workers

All workers declare (or connect to) the same durable queue. Each worker consumes with `prefetch=1` and manual ack. RabbitMQ distributes messages round-robin across workers. If a worker crashes before acking, RabbitMQ re-delivers the message to another worker.

Spring AMQP — competing consumers with manual ack

// Worker (same code, deployed N times)
@Component
class OrderWorker {

    @RabbitListener(
        queues = "order-processing",
        containerFactory = "workerContainerFactory"
    )
    public void process(Order order, Channel channel,
                        @Header(AmqpHeaders.DELIVERY_TAG) long tag) {
        try {
            orderService.fulfil(order);
            channel.basicAck(tag, false);           // done
        } catch (RecoverableException e) {
            channel.basicNack(tag, false, true);    // requeue for retry
        } catch (Exception e) {
            channel.basicNack(tag, false, false);   // → DLX, no requeue
        }
    }
}

@Bean
SimpleRabbitListenerContainerFactory workerContainerFactory(ConnectionFactory cf) {
    var f = new SimpleRabbitListenerContainerFactory();
    f.setConnectionFactory(cf);
    f.setPrefetchCount(1);                          // fair dispatch
    f.setAcknowledgeMode(AcknowledgeMode.MANUAL);
    return f;
}

Scaling Up and Down with Kubernetes

Deploy the worker as a Kubernetes Deployment and scale the `replicas` field. KEDA (Kubernetes Event-Driven Autoscaling) can automatically scale the deployment based on RabbitMQ queue depth — scale up when the queue grows, scale down when it drains. Use `minReplicaCount=1` to keep at least one worker running.

Kubernetes + KEDA — autoscale workers on queue depth

# Kubernetes Deployment — scale replicas to match load
apiVersion: apps/v1
kind: Deployment
metadata: { name: order-worker }
spec:
  replicas: 3                         # horizontal scaling
  template:
    spec:
      containers:
        - name: worker
          image: order-worker:1.2.0
          env:
            - name: SPRING_RABBITMQ_HOST
              value: rabbitmq-service

---
# KEDA ScaledObject — auto-scale based on queue depth
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: order-worker-scaler }
spec:
  scaleTargetRef: { name: order-worker }
  minReplicaCount: 1
  maxReplicaCount: 20
  triggers:
    - type: rabbitmq
      metadata:
        queueName: order-processing
        queueLength: "50"              # 1 replica per 50 messages

Message Ordering and Idempotency Considerations

Competing consumers break strict message ordering — different workers may process messages concurrently out of order. If ordering matters (e.g., account state transitions), use a **single consumer** or **consistent hashing exchange** (community plugin) to route all messages for the same entity to the same worker. Always design workers to be **idempotent**: if a message is re-delivered (worker crash before ack), processing it twice should be safe.

Spring AMQP — idempotent worker with dedup table

// Idempotent worker — safe to re-process
@RabbitListener(queues = "order-processing")
public void process(Order order, Channel ch,
                    @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
    // Check if already processed (using DB or Redis dedup key)
    if (processedMessageRepo.existsById(order.getMessageId())) {
        channel.basicAck(tag, false);   // already done — ack and skip
        return;
    }
    try {
        orderService.fulfil(order);
        processedMessageRepo.save(new ProcessedMessage(order.getMessageId()));
        channel.basicAck(tag, false);
    } catch (Exception e) {
        channel.basicNack(tag, false, false);   // → DLX
    }
}

Key Points to Remember

1Competing consumers deliver each message to exactly one worker — natural horizontal scaling
2prefetch=1 + manual ack ensures fair dispatch; workers only receive work they can handle
3RabbitMQ round-robins messages across consumers; slow workers won't block fast ones with prefetch=1
4Crashed workers: unacked messages are re-queued and delivered to another worker
5Competing consumers break message ordering — use consistent hashing if order matters
6Design workers to be idempotent — re-delivery after crash must be safe to re-process

Interview Questions

How does the Competing Consumers pattern provide horizontal scaling in RabbitMQ?

EasyAmazon

Why is prefetch=1 important for fair work distribution in a worker queue?

MediumDeliveroo

What happens to unacknowledged messages when a consumer crashes?

EasyBooking.com

How would you maintain message ordering while still scaling consumer count?

HardZalando

Why must workers in a competing-consumers setup be idempotent?

MediumThoughtWorks

Ask Aria about Competing Consumers Pattern

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Request-Reply (RPC) Pattern

RabbitMQ with Spring AMQP