Home/Learn/RabbitMQ/RabbitMQ Performance Tuning

RabbitMQ Performance Tuning

Advanced

Operations

Tune prefetch count, connection/channel pooling, persistent vs transient messages, queue type selection, and OS TCP settings to maximise throughput and minimise latency.

Overview

RabbitMQ performance is determined by several interacting factors: message persistence (disk I/O vs memory), prefetch count (consumer parallelism and backpressure), connection/channel management (TCP overhead), queue type (classic vs quorum vs stream), and broker resource limits (memory high watermark, disk free limit). The most impactful single tuning is prefetch count: too high floods slow consumers; too low underutilises fast consumers. Persistent messages (delivery-mode=2) + durable queues guarantee durability at a throughput cost. Transient messages (delivery-mode=1) + non-durable queues maximise throughput when some loss is acceptable. RabbitMQ exposes per-queue and per-connection metrics via the management plugin for performance analysis.

Prefetch count tuning for consumer throughput

Prefetch count (basic.qos) controls how many unacknowledged messages a consumer can hold at once. prefetch=1 ensures fair dispatch (slow consumers don't accumulate backlog) but limits throughput. Large prefetch values increase throughput but can cause uneven distribution and memory pressure on slow consumers. For most workloads, start with prefetch=10–50 and tune based on consumer processing time and message size.

Java — prefetch count and concurrent consumer configuration

// Spring AMQP — configure prefetch count
@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
        ConnectionFactory connectionFactory) {
    SimpleRabbitListenerContainerFactory factory =
        new SimpleRabbitListenerContainerFactory();
    factory.setConnectionFactory(connectionFactory);

    // Prefetch: messages pre-fetched per consumer
    factory.setPrefetchCount(10);     // default 250 in Spring AMQP

    // Concurrent consumers: multiple threads per container
    factory.setConcurrentConsumers(3);
    factory.setMaxConcurrentConsumers(10);  // scale up under load

    // Acknowledgement mode
    factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);

    return factory;
}

// @RabbitListener automatically uses the factory
@RabbitListener(queues = "order.processing")
public void processOrder(OrderMessage msg, Channel channel,
                         @Header(AmqpHeaders.DELIVERY_TAG) long tag)
        throws IOException {
    try {
        processOrder(msg);
        channel.basicAck(tag, false);
    } catch (Exception e) {
        channel.basicNack(tag, false, true);  // requeue
    }
}

Message persistence and transient messages

Durable queues + persistent messages (delivery-mode=2) survive broker restart but require disk writes on every message, reducing throughput. Transient messages (delivery-mode=1) on non-durable queues stay in memory — 10–50x higher throughput but lost on restart. Choose based on message criticality: payments and order events must be persistent; telemetry and notifications can be transient.

Java — persistent vs transient messages and queue type selection

// Spring AMQP — persistent message (default for convertAndSend)
rabbitTemplate.convertAndSend(exchange, routingKey, message, msg -> {
    msg.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
    return msg;
});

// Transient message — stays in memory, much faster
rabbitTemplate.convertAndSend(exchange, routingKey, message, msg -> {
    msg.getMessageProperties().setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT);
    return msg;
});

// Queue type selection — performance characteristics:
// Classic (default):   single-node, in-memory first, ~50k msg/s
// Quorum (recommended): replicated, disk-based, ~20k msg/s, survives node loss
// Stream:              append-only log, millions msg/s, replay capable

@Bean
public Queue performanceQueue() {
    return QueueBuilder
        .durable("high-throughput.stream")
        .stream()           // Kafka-like stream queue
        .build();
}

@Bean
public Queue reliableQueue() {
    return QueueBuilder
        .durable("payments.quorum")
        .quorum()           // replicated quorum queue
        .build();
}

Connection pooling and broker memory management

Each AMQP connection uses a TCP socket and ~50 KB memory on the broker. Channels are lightweight virtual connections multiplexed over TCP — use multiple channels per connection rather than multiple connections. Set the memory high watermark (0.4 = 40% of RAM) to prevent the broker from running out of memory under load. The disk free limit prevents writes from filling the disk.

Config + Java — broker memory limits and connection pool configuration

# RabbitMQ broker configuration (rabbitmq.conf)

# Memory high watermark — when reached, broker blocks all producers
vm_memory_high_watermark.relative = 0.4   # 40% of RAM (default 0.4)
# Or absolute:
# vm_memory_high_watermark.absolute = 4GB

# Disk free limit — block when disk space drops below threshold
disk_free_limit.absolute = 2GB   # keep at least 2 GB free

# Connection and channel limits
channel_max = 2047               # channels per connection
connection_max = 1000            # total connections

# Lazy queues (classic) — write to disk immediately, low memory pressure
queue_master_locator = min-masters  # balance queue masters across nodes

# Spring AMQP — reuse connections via CachingConnectionFactory
@Bean
public CachingConnectionFactory connectionFactory() {
    CachingConnectionFactory factory = new CachingConnectionFactory("rabbitmq");
    factory.setChannelCacheSize(25);       // cache 25 channels per connection
    factory.setConnectionCacheSize(2);     // maintain 2 connections
    factory.setCacheMode(CachingConnectionFactory.CacheMode.CHANNEL);
    return factory;
}

Key Points to Remember

1Prefetch count = 1 ensures fair dispatch but limits throughput; start at 10–50 and tune based on consumer processing time
2Persistent messages (delivery-mode=2) + durable queues survive restart but require disk writes — use for business-critical events
3Transient messages on non-durable queues are 10–50x faster but lost on restart — acceptable for telemetry, not payments
4Quorum queues are the recommended default for reliability (replicated, survives node failure); Stream queues for high-throughput replay
5vm_memory_high_watermark=0.4 blocks producers when broker reaches 40% RAM — prevents OOM broker crashes
6Channels are multiplexed over TCP connections — use CachingConnectionFactory channel cache instead of creating new connections

Interview Questions

What is prefetch count and how does it affect consumer throughput and fairness?

MediumThoughtworks

When would you use transient messages instead of persistent messages in RabbitMQ?

EasyInfosys

What is the RabbitMQ memory high watermark and what happens when it is reached?

MediumAmazon

Compare Classic, Quorum, and Stream queue types — when would you choose each?

HardNetflix

You have 500 Spring Boot instances each opening their own connection to RabbitMQ. What problem does this cause and how do you fix it?

HardUber

Ask Aria about RabbitMQ Performance Tuning

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Monitoring & Alerting

RabbitMQ in Microservices