RabbitMQ — Cheat Sheet

To decouple producers from consumers in time, in load, and in identity. In time: the producer does not wait for the consumer. An order can be accepted and acknowledged while fulfilment happens later, so a slow downstream does not inflate the user-facing response. In load: the queue absorbs bursts. A traffic spike that would overwhelm a synchronous downstream becomes a queue that drains at whatever rate the consumer can manage. That is back-pressure, and it converts a failure into a delay. In identity: the producer does not know who consumes. Adding a second consumer for a new feature requires no change to the publisher, which is the main architectural benefit — it stops every new requirement rippling backward. It also provides durability: if the consumer is down, messages wait rather than being lost. The costs are real and worth stating. You have added a component to operate and monitor. Debugging becomes harder because flow is indirect. Ordering and exactly-once semantics become your problem. And eventual consistency means clients can read state that does not yet reflect their own write. So a broker is right when the coupling it removes is genuinely a problem, not by default.

AMQP is a wire-level messaging protocol — it specifies the byte format on the connection, not just an API — which means clients from different vendors can interoperate. RabbitMQ was built as an AMQP 0-9-1 broker, and that version is what most people mean when they say AMQP in a Rabbit context. It defines the exchange, queue and binding model that gives RabbitMQ its routing flexibility. AMQP 1.0 is a substantially different protocol despite the name — it dropped the exchange model and is more of a peer-to-peer transport. RabbitMQ supports it via a plugin, but it is not the native model. RabbitMQ also speaks other protocols through plugins: MQTT for IoT, STOMP for simple text-based clients, and its own stream protocol for the streams feature. The practical relevance of the wire-level point is that you are not locked to a client library — any AMQP 0-9-1 client works, and the semantics are defined by the protocol rather than by a particular SDK. It also explains why RabbitMQ's model has concepts other brokers lack: exchanges and bindings come from AMQP, not from Rabbit's own design.

A queue delivers each message to exactly one consumer — competing consumers share the work. A topic delivers each message to every interested subscriber — publish-subscribe. The distinction is about whether you are distributing work or broadcasting information. RabbitMQ expresses both through the same primitives rather than having separate types. Messages always go to an exchange, which routes them to zero or more queues. Consumers always read from queues. So work distribution is several consumers on one queue, and pub-sub is several queues bound to one exchange, each with its own consumer. The exchange type controls the routing. That indirection is RabbitMQ's main structural difference from brokers where you publish directly to a queue or a topic. It is more flexible — you can change routing without touching publishers — at the cost of more concepts to learn. The practical consequence worth knowing: if two consumers should each receive every message, they need separate queues. Putting them on the same queue makes them competing consumers and each message goes to only one, which is a very common early mistake.

They solve different problems despite both being called message systems. RabbitMQ is a message broker built for routing and per-message delivery. It excels at complex routing through exchanges, per-message acknowledgement and redelivery, priority queues, delayed delivery, and request-reply patterns. Messages are removed once consumed. Kafka is a distributed log. It excels at very high throughput, retaining messages so they can be replayed, ordered partitions, and many independent consumer groups reading the same stream at their own pace. Choose RabbitMQ for task distribution and command-style messaging where each message is work to be done once, where routing logic is genuinely complex, and where you want the broker to track per-message state. Choose Kafka for event streaming, for cases where replay matters, for very high volume, and where multiple consumers need the same events independently over time. The practical discriminator: if you find yourself wanting to reprocess last week's messages, you want a log. If you want a message to disappear once handled and to be redelivered on failure, you want a broker. Many systems legitimately run both.

A command tells a specific service to do something — PlaceOrder, ChargeCard. It has one intended handler, expects to be acted on, and the sender knows who should handle it. An event announces that something happened — OrderPlaced, PaymentReceived. It is past tense, has any number of interested consumers, and the publisher neither knows nor cares who reacts. The design consequence is coupling. Commands couple the sender to the receiver: adding a new step means changing the sender. Events do not: adding a consumer requires no change to the publisher, which is what makes event-driven architectures extensible. It also affects delivery semantics. A command going unhandled is a failure. An event with no consumers is perfectly fine. In RabbitMQ terms, a command typically goes to a direct exchange with one bound queue. An event goes to a fanout or topic exchange with a queue per interested consumer. The common design mistake is publishing what is really a command as an event — naming it in past tense but expecting exactly one specific consumer to act. That gets you the ambiguity of events with the coupling of commands.

At-most-once: a message may be lost but never duplicated. Achieved by acknowledging before processing, or by not acknowledging at all. At-least-once: a message is never lost but may be delivered more than once. Achieved by acknowledging only after successful processing, so a crash before the ack causes redelivery. Exactly-once: delivered precisely once. Not achievable in a distributed system in the general case, because the acknowledgement itself can be lost — the consumer cannot distinguish "the message was not processed" from "it was processed and my ack was lost". RabbitMQ provides at-least-once when configured correctly: durable queues, persistent messages, publisher confirms, and manual consumer acknowledgement after processing. The practical answer to the exactly-once question is idempotent consumers. If processing the same message twice has the same effect as once, at-least-once delivery is functionally equivalent to exactly-once from the application's point of view. That shifts the problem from the broker to the consumer, which is where it can actually be solved — usually with a deduplication key recorded atomically with the effect.

Back-pressure is a signal from a slow component telling a fast one to slow down, rather than being overwhelmed. A queue provides it implicitly by absorbing the difference between production and consumption rate. Short bursts are buffered and drained. The crucial part is what happens when the buffer is not enough. An unbounded queue provides no back-pressure at all — it grows until memory or disk is exhausted, converting a throughput problem into an outage. That is the failure people do not anticipate, because the queue looks like it is working right up until it is not. A bounded queue with a max-length policy provides real back-pressure: when full it either rejects new messages or drops old ones, and the publisher learns about it. RabbitMQ also has flow control at the connection level — when the broker is under memory or disk pressure it blocks publishing connections, which pushes back to producers directly. The design guidance is to always bound queues and decide the overflow behaviour deliberately: reject and let the publisher handle it, drop the oldest for time-sensitive data, or dead-letter for later inspection. Leaving it unbounded is choosing to fail catastrophically instead of gracefully.

You need to update the database and publish a message, and there is no transaction spanning both systems. Publish first and the transaction may fail to commit, so consumers act on something that did not happen — a shipment for an order that does not exist. Commit first and the publish may fail, so the event is lost and downstream systems never learn about it. Both failure modes occur in practice, and they are not rare — a crash between the two operations is enough. The standard solution is the transactional outbox. Write the message to an outbox table in the same database transaction as the state change, so atomicity comes from the database. A separate relay process reads unpublished rows, publishes them, and marks them sent. The relay can poll the table or use change data capture reading the transaction log, which avoids polling load — Debezium is the common tool. Because the relay can crash after publishing and before marking, delivery is at-least-once, so consumers must be idempotent. The naive alternative — publishing inside the service method and hoping — is extremely common and is a latent correctness bug rather than a design.

Processing the same message twice produces the same result as processing it once. That is what makes at-least-once delivery safe. The general mechanism is a deduplication key. Each message carries a unique identifier; the consumer records processed identifiers and skips repeats. The critical detail is atomicity. Checking whether the ID was seen and then performing the work is a race — two concurrent redeliveries can both pass the check. The ID must be recorded in the same transaction as the effect, or inserted first with a unique constraint so the duplicate fails cleanly. Better still is to design operations that are naturally idempotent. Setting a status to shipped is idempotent; appending to a list is not. Upserting a row keyed by the message ID is; inserting is not. Reframing an operation as an assignment often removes the need for a dedup table entirely. For external side effects — sending an email, charging a card — idempotency must be pushed to that system too, usually via an idempotency key it supports. And decide how long to retain keys: forever is a growth problem, too short reopens the window.

Only within a single queue with a single consumer, and even then only if you never requeue. Messages published to one queue by one publisher are enqueued in order, and a single consumer receives them in order. That is the guarantee. It breaks in several common situations. Multiple consumers on a queue process concurrently, so completion order is not delivery order. A prefetch greater than one with a single consumer still delivers in order but allows concurrent processing if the consumer is multithreaded. And requeuing a failed message puts it back — historically at the head, which changes relative order. Multiple publishers have no defined interleaving. And a message routed to several queues has no cross-queue ordering. So if you need strict ordering, you need one queue, one consumer, prefetch of one, and no requeue-on-failure — which sacrifices throughput entirely. The better design is usually to avoid needing global order. Partition by an entity key so that messages for one order are ordered relative to each other while different orders proceed in parallel. RabbitMQ has consistent hash exchanges for this; Kafka does it natively with partitions. Or make handlers commutative so order does not matter.

A connection is a TCP connection to the broker. A channel is a lightweight logical session multiplexed over that connection. The reason for the distinction is cost. TCP connections are expensive to establish and each consumes a file descriptor and memory on the broker. An application with fifty threads should not open fifty connections. Instead it opens one connection and fifty channels, which multiplex over it. Channels are cheap to create and destroy. The rules that matter in practice: channels are not thread-safe, so each thread needs its own. Sharing a channel across threads produces corrupted framing and confusing errors, and it is one of the most common client bugs. A channel that encounters an error is closed by the broker, so your client must handle channel closure and recreate — many clients do this automatically but it is worth knowing. Publishing and consuming on the same channel is allowed but can interact badly under flow control, so separate channels for each is the usual recommendation. And very high channel counts on one connection eventually contend on the single TCP stream, so extremely high throughput may warrant several connections.

A virtual host is a namespace within a broker containing its own exchanges, queues, bindings and permissions. Names are scoped to it, so two vhosts can each have a queue called orders with no relationship. The purpose is logical isolation on shared infrastructure. Separate applications, or separate environments, can share a broker without seeing or colliding with each other. Permissions are granted per vhost, so a user with access to one cannot touch another — which makes it a genuine security boundary, not just an organisational one. The limits are worth knowing. It is not a performance boundary: vhosts share the broker's memory, disk and CPU, so a runaway queue in one affects all of them. It is also not a fault boundary — a broker problem affects everything. So vhosts are appropriate for separating applications or teams on a shared cluster where the isolation needed is naming and access control. They are not a substitute for separate clusters when you need resource or failure isolation. The default vhost is "/" and every connection specifies one. Using the default for everything works but loses the separation, and retrofitting vhosts later means reconfiguring every client.

Direct routes by exact match between the message's routing key and the binding key. Use it for point-to-point work distribution where each message has one clear destination — a task queue keyed by job type. Fanout ignores the routing key entirely and copies the message to every bound queue. Use it for broadcast: an event that several independent services all need. Topic routes by pattern, with * matching one word and # matching zero or more, on dot-separated routing keys. order.created.in matches order.*.in and order.#. This is the most flexible and the usual choice for event-driven systems, because consumers subscribe to the slice they care about without the publisher knowing. Headers routes on message header values rather than the routing key, matching all or any of a set. Rarely used — it is slower and topic exchanges cover most cases more readably. The default exchange is a direct exchange with an empty name, to which every queue is automatically bound by its own name. That is why publishing with a routing key equal to a queue name appears to go straight to the queue, which confuses people learning the model.

The routing key is a dot-separated string, conventionally from general to specific: order.created.in, payment.failed.uk. Binding keys use two wildcards. An asterisk matches exactly one word. A hash matches zero or more words. So order.* matches order.created but not order.created.in. order.# matches order, order.created and order.created.in. And *.created.* matches order.created.in but not order.created. A queue bound with # receives everything, which makes fanout expressible as a topic binding. The design work is in the routing key scheme, and it is worth thinking about before the first publisher ships. A good scheme puts the most commonly filtered dimension early, because that is what the wildcards can select on. Putting the entity type first and the action second — order.created rather than created.order — lets a consumer subscribe to all order events with one binding. Adding a dimension later is awkward, since existing bindings with a fixed number of words stop matching when you append a segment. Bindings ending in # are resilient to that; ones ending in * are not. That is a genuine reason to end bindings with # where you can.

They are two separate settings and you need both for messages to survive a broker restart. This is the classic gotcha. A durable queue survives broker restart — its definition is written to disk, so the queue still exists when the broker comes back. A non-durable queue is gone. A persistent message is written to disk. A transient message lives only in memory. So a persistent message in a non-durable queue is lost, because the queue itself disappears. A transient message in a durable queue is lost too, because only the queue definition was saved. Only durable plus persistent survives. The cost is disk I/O on every publish, which reduces throughput substantially compared to transient messaging. The caveat worth raising is that persistence is not an absolute guarantee. A message is written to disk asynchronously, so there is a brief window between the broker accepting it and it reaching disk. A crash in that window loses it despite both flags. Publisher confirms close that gap, because the broker only confirms after the message is safely persisted. So durable plus persistent plus confirms is what actually gives the guarantee people assume they get from durable alone.

Quorum queues replicate using the Raft consensus algorithm, with a leader and followers agreeing on the log of messages. A message is confirmed once a majority has it. They replaced classic mirrored queues because mirroring had genuine correctness problems. Mirrored queues used a leader-follower model without consensus, and under network partitions they could lose confirmed messages during a failover — the new leader might not have everything the old one had acknowledged. The synchronisation behaviour was also operationally awkward, with full resynchronisation of a large queue blocking the queue entirely. Quorum queues give a much stronger guarantee: a confirmed message survives the loss of a minority of nodes, and failover does not lose acknowledged data. The trade-offs. They need an odd number of replicas, typically three or five, and a majority available to accept writes — so a three-node cluster tolerates one failure. They use more memory and disk since every message is replicated. And they do not support some features of classic queues: priorities were unsupported initially, and message TTL behaviour differs. Mirrored queues are deprecated and removed in RabbitMQ 4. Quorum queues are the default recommendation for anything requiring durability.

Streams are an append-only log, added in RabbitMQ 3.9, that behaves much more like Kafka than like a traditional queue. The key difference is that consuming does not remove. A stream retains messages according to a size or age policy, and consumers track their own offset. Several consumers can read the same stream independently, and a consumer can rewind and replay. That solves the main thing classic queues cannot do: reprocessing history. With a queue, once a message is acknowledged it is gone, so a bug in a consumer means the data is unrecoverable. Streams also handle very high throughput better, because the append-only structure and a dedicated binary protocol avoid the per-message bookkeeping queues require. The trade-offs: no complex routing to a stream in the same way, no per-message acknowledgement and redelivery semantics, and no priorities. Non-destructive reads mean the broker cannot tell you what is outstanding per consumer in the same way. So the choice mirrors the RabbitMQ-versus-Kafka question within one broker: use a queue for work distribution where each message is handled once, and a stream for event history that multiple consumers read and may need to replay.

A lazy queue writes messages to disk as soon as possible and keeps as little as possible in memory, rather than holding messages in RAM and only paging out under pressure. The motivation is predictability with large backlogs. A default queue holding a million messages consumes a lot of memory, and when the broker hits its memory watermark it pages messages to disk in a burst — which blocks the queue and causes a latency spike exactly when you are already under stress. A lazy queue avoids that cliff by never building the in-memory backlog in the first place. Throughput is lower in the steady state because every message touches disk, but behaviour under a large backlog is far more stable. So the use case is queues that are expected to accumulate — a consumer that goes down for maintenance, batch processing, or any queue where depth is measured in hundreds of thousands. For low-latency queues that stay near empty, the default mode is faster. In RabbitMQ 3.12 and later the distinction largely disappeared: classic queues adopted a single version-2 storage implementation that behaves lazily by default, so the explicit setting is deprecated.

Three levels, and they interact. A per-queue message TTL, set with x-message-ttl on the queue, expires any message that sits in that queue longer than the limit. A per-message TTL, set in the message's expiration property, applies to that message alone. When both are set the shorter wins. Separately, a queue TTL with x-expires deletes the entire queue after a period of being unused — no consumers and no activity. That is useful for temporary reply queues that would otherwise accumulate. The important behaviour is what happens to an expired message: if the queue has a dead-letter exchange configured, expired messages are dead-lettered rather than discarded, which is how you build a delayed-message mechanism. The gotcha with per-queue TTL is that only the message at the head is checked for expiry in classic queues. A message behind a long-lived one may sit past its TTL until it reaches the head. So TTL is a lower bound on removal, not a precise timer. And changing x-message-ttl requires deleting and recreating the queue, since queue arguments are immutable.

Two approaches. The TTL plus dead-letter trick: publish to a holding queue that has a message TTL and a dead-letter exchange pointing at the real queue. The message sits for the TTL, expires, and is dead-lettered into the destination — effectively a delay. It works with no plugins, but it has a significant limitation. Because classic queues only check the message at the head for expiry, messages with different delays in the same queue block each other. A message with a ten-minute delay ahead of one with a one-minute delay means the second waits ten minutes. So this pattern needs a separate holding queue per delay value, which does not scale to arbitrary delays. The delayed message exchange plugin is the better answer for variable delays. It holds messages in the exchange itself with a per-message delay header and routes them when due. Its caveats: it is a community plugin, the delayed messages are held in the exchange rather than a replicated queue so high availability is weaker, and very large numbers of delayed messages consume broker resources. For long or precise scheduling, an external scheduler writing to the queue when due is often more robust.

A queue declared with x-max-priority delivers higher-priority messages before lower ones. Each message carries a priority in its properties. The limitations are substantial enough that it is often the wrong tool. Priority only applies to messages waiting in the queue. If consumers keep up and the queue is near empty, there is nothing to reorder and priority has no effect — which surprises people who add it and see no change. It only works within a queue, so a low-priority message already delivered to a consumer is not preempted. Each priority level costs resources — RabbitMQ maintains internal sub-queues — so a large number of levels is expensive. The documentation recommends keeping it to a handful, not the full 0-255 range. Quorum queues did not originally support priorities at all, which matters if you also want replication. And starvation is a real risk: a steady stream of high-priority messages means low-priority ones never run. The usual better design is separate queues per priority class with dedicated consumers, or more consumers on the urgent queue. That gives explicit capacity allocation rather than relying on reordering.

An alternate exchange receives messages that an exchange could not route to any queue. Without one, an unroutable message is silently dropped — unless the publisher set the mandatory flag and handles the return. Silent dropping is the default, and it is how messages disappear with no error and no trace. Configuring an alternate exchange on the primary exchange means those messages go somewhere you can inspect. Typically it is a fanout exchange bound to an unroutable queue, which you monitor and alert on. The value is diagnostic. A non-empty unroutable queue tells you a publisher is using a routing key nothing is bound to — usually a typo, a missing binding after a deployment, or a consumer that was decommissioned while publishers still send to it. Without it, that condition is invisible until someone notices missing data downstream, which can be days later. The alternative mechanism is the mandatory flag with a return listener on the publisher, which surfaces the problem at the point of publish. That is arguably better because the publisher learns immediately, but it requires every publisher to implement the handler. Using both is reasonable: mandatory for immediate feedback, alternate exchange as the safety net.

The general rule is that the consumer declares its own queue and binding, and the publisher declares only the exchange. The reasoning: the exchange is the contract the publisher depends on, so it must exist for publishing to work. The queue is the consumer's private concern — it decides what it wants to receive and binds accordingly. A publisher should not know which queues exist, because that is exactly the coupling the exchange model removes. Declarations are idempotent, so both sides declaring the exchange is harmless — provided the arguments match. Declaring with different arguments than an existing entity fails with a channel error, which is a common startup problem after someone changes a queue setting. That immutability is worth stressing: queue arguments cannot be changed in place. Adding a TTL or a dead-letter exchange to an existing queue means deleting and recreating it, which loses messages unless handled carefully. The alternative approach, used in more controlled environments, is to declare all topology through infrastructure-as-code and have applications declare nothing — connecting to entities that must already exist. That gives review and versioning of topology changes, at the cost of a deployment dependency.

An exclusive queue is usable only by the connection that declared it, and is deleted when that connection closes. An auto-delete queue is deleted when its last consumer disconnects. The main use is temporary queues that belong to one client instance. A request-reply pattern where each client needs its own reply queue, or a subscriber that wants a private feed of events for as long as it is running. Using them means you do not accumulate abandoned queues when clients come and go, which otherwise becomes a real operational problem — thousands of orphaned queues each consuming resources. The distinction matters: exclusive dies with the connection even if nobody ever consumed; auto-delete dies when consumers leave, and a queue that never had a consumer is not auto-deleted. The caution is that these queues are inherently non-durable in effect — a client reconnect creates a new queue, and any messages routed while it was gone are lost. So they suit ephemeral subscriptions and not work that must not be missed. For reply queues specifically, RabbitMQ offers direct reply-to, a pseudo-queue that avoids creating a real queue per request at all, which is more efficient for high-volume RPC.

A binding is a rule connecting an exchange to a destination, with an optional binding key that the exchange type interprets. A queue can be bound to several exchanges, and to the same exchange several times with different keys — so one queue can receive order.created and payment.failed by having two bindings. Exchange-to-exchange bindings are supported and are genuinely useful. Messages routed to the first exchange are forwarded to the second, which then applies its own routing. The use is composing routing topologies. A common pattern is a single ingress exchange that every publisher knows, bound to several downstream exchanges owned by different domains. Publishers depend on one stable name, while the routing behind it can be reorganised without touching them. It also lets you insert a layer — for instance an exchange that fans out to both the live consumers and an archival queue — without changing publishers or existing bindings. The caution is that deep chains make routing hard to reason about, and a message's path becomes non-obvious. The management UI can trace a routing key against the topology, which is the tool to reach for when a message does not arrive where expected.

By default it is silently discarded. The publish succeeds, the broker accepts the message, and it goes nowhere. That is the single most common source of "my messages are disappearing" — a routing key with a typo, a binding that was never created, or a queue deleted while publishers still send. There are three ways to make it visible. The mandatory flag on publish tells the broker to return the message to the publisher if it cannot be routed. The client receives it via a return listener, so the publisher learns immediately and can log or handle it. This requires implementing the listener, which many applications skip. An alternate exchange on the exchange catches unroutable messages and sends them to a queue you can monitor. This works regardless of publisher implementation and is the better safety net. Publisher confirms alone do not help here — a confirm means the broker accepted the message, not that it was routed anywhere. That distinction catches people out: you can have confirms enabled, receive an ack, and still lose the message. Use mandatory plus an alternate exchange, and alert on the unroutable queue depth.

Publisher confirms are an acknowledgement from the broker that it has taken responsibility for a message. Without them, a publish is fire-and-forget — the client hands the message to the socket and continues, with no indication of whether the broker received it. That matters because a broker crash, a network failure, or a full disk can all cause a message to be lost with the publisher believing it succeeded. With confirms enabled on a channel, the broker sends an ack once the message is safely handled — for a persistent message in a durable queue, that means after it has been written to disk. A nack means the broker could not take it. The modes: waiting for each confirm individually is simple and slow. Publishing in batches and waiting for the batch is a reasonable compromise. Handling confirms asynchronously with a callback gives the best throughput but requires tracking outstanding sequence numbers so you know which message a nack refers to. The crucial limitation to state: a confirm means the broker has the message, not that it was routed to a queue. An unroutable message is confirmed and then discarded. Mandatory publishing covers that gap.

With automatic acknowledgement, a message is considered delivered the moment the broker sends it. The broker removes it immediately and moves on. With manual acknowledgement, the consumer explicitly acks after processing, and the broker retains the message until then. If the consumer disconnects without acking, the message is requeued and redelivered. Automatic ack gives at-most-once semantics — a consumer crash after receipt and before processing loses the message permanently. It is faster because there is no round trip and no per-message tracking, and the broker applies no flow control, so it will push messages as fast as the socket allows. That last point is a real hazard: with auto-ack the broker ignores prefetch, so a slow consumer can be sent an unbounded number of messages and run out of memory. Manual ack gives at-least-once, which is what almost every application actually wants. So the guidance is manual acknowledgement with a sensible prefetch as the default, and auto-ack only for genuinely disposable data such as metrics or logs where loss is acceptable and throughput matters more.

Prefetch — basic.qos — limits how many unacknowledged messages the broker will send to a consumer at once. Once that many are outstanding, the broker stops sending until some are acknowledged. Without it the broker pushes everything it has to the first available consumer, which causes two problems: that consumer may run out of memory, and work is distributed badly because one consumer holds a large batch while others idle. The value is a trade-off. Prefetch of 1 gives the fairest distribution — each consumer takes the next message only when free — at the cost of a network round trip per message, which caps throughput. Very useful when message processing times vary a lot, since one slow message does not block a batch. A higher prefetch amortises the round trip and increases throughput, but a consumer holding many messages that then dies causes all of them to be redelivered, and distribution becomes lumpier. The rule of thumb is to start low — often 1 to 10 for slow tasks — and raise it for fast, uniform messages. The RabbitMQ documentation suggests computing from round-trip time and processing time. The common mistake is leaving it unlimited.

basic.reject rejects a single message. basic.nack is a RabbitMQ extension that can reject multiple messages at once with the multiple flag. Otherwise they behave the same. Both take a requeue flag, and that flag is the important decision. With requeue true, the message goes back to the queue to be delivered again. With requeue false, it is either dead-lettered if a dead-letter exchange is configured, or discarded if not. The trap is requeuing on a failure that will always fail. A malformed message that throws on parse, requeued, is redelivered immediately, fails again, and loops — a poison message spinning at full speed, consuming CPU and filling logs. This is one of the most common RabbitMQ production incidents. So the rule is: requeue only for transient failures where a retry has a chance — a temporarily unavailable downstream. For anything deterministic, reject without requeue so it dead-letters. And even for transient failures, unbounded requeue is dangerous. Track a retry count in a header and dead-letter after a threshold, so a persistent outage does not produce an infinite loop.

Several consumers subscribe to the same queue, and the broker delivers each message to one of them. That is the work-distribution pattern, and scaling consumers scales throughput. Distribution is round-robin by default, bounded by prefetch. With prefetch of 1, the broker gives each consumer one message and waits for the ack before sending another — so a slow consumer naturally receives less work, which is what you usually want. With a high prefetch, the broker hands out batches in advance, so a consumer that turns out to be slow is holding messages that others could have processed. That is how you get one consumer backed up while others idle, and the fix is a lower prefetch. The consequences to be aware of: ordering is lost, since concurrent consumers finish in arbitrary order. And a message being processed is invisible to other consumers but is not lost — if that consumer dies without acking, it is redelivered elsewhere. Single active consumer is an option when you need ordering: multiple consumers connect but only one receives messages, with automatic failover to another if it disconnects. That gives ordered processing with high availability, at the cost of no parallelism.

The client publishes a request with two properties set: reply_to naming a queue for the response, and correlation_id identifying the request. The server processes and publishes the response to the reply_to queue, echoing the correlation_id. The client matches responses to outstanding requests by that identifier — necessary because responses can arrive out of order and one client may have many requests in flight. For the reply queue, the naive approach is an exclusive auto-delete queue per client, which works but costs a queue declaration per client. Direct reply-to is the better mechanism: a pseudo-queue that requires no declaration and no cleanup, designed for exactly this. The design caution worth raising is whether you should be doing this at all. Request-reply over a broker gives you the latency and coupling of a synchronous call plus the operational complexity of a broker. If the caller waits for the answer, an HTTP or gRPC call is usually simpler and easier to debug. It earns its place when you want the broker's load balancing across many workers, when the backend is not reachable directly, or when you need the request to survive a consumer restart. Always set a timeout on the client side.

delivery_mode set to 2 makes the message persistent — the single most important one, and easily forgotten. content_type so consumers know how to deserialise, which matters when a queue carries more than one format or when you migrate serialisation. message_id as a unique identifier, which is what consumer-side deduplication keys on. Without it, idempotency is much harder. correlation_id to tie a message to a request or a saga, and to trace a chain of related messages across services. timestamp for when the message was created, which is essential for measuring end-to-end latency and for spotting messages that have been sitting too long. headers for anything application-specific: a schema version, a retry count, a trace context. Type or a header naming the event type, so a consumer reading from a queue with several event types can dispatch without inspecting the body. The two most commonly omitted and most missed are message_id and a schema version. The first blocks deduplication; the second makes evolving the payload format much harder, because consumers cannot tell which shape they received.

JSON is the pragmatic default: human-readable, universally supported, easy to debug from the management UI, and tolerant of unknown fields if consumers are written to ignore them. The costs are size and the absence of a schema, so nothing prevents a publisher changing the shape and breaking consumers silently. Binary formats — Protobuf, Avro — give compact payloads and, crucially, an explicit schema with defined compatibility rules. Avro with a schema registry is the standard approach where message contracts matter, because the registry can reject an incompatible schema before it is ever published. The cost is that messages are opaque in the management UI, and every consumer needs the schema. Whatever the format, include a version. A schema version header lets a consumer handle several shapes during a rollout, which is what makes zero-downtime deployment possible when the payload changes. And apply the same compatibility discipline as an API: add optional fields, never remove or rename, never change a field's type or meaning. Consumers must ignore unknown fields, and that expectation should be documented from the first release since it cannot be retrofitted.

basic.qos takes a global flag that changes the scope of the limit. With global false — the default in most clients — the prefetch count applies per consumer. A channel with three consumers and a prefetch of 10 allows up to 30 unacknowledged messages, 10 per consumer. With global true, the limit applies to the whole channel, so all consumers on it share the 10. The per-consumer form is usually what you want, because it means adding a consumer adds capacity rather than dividing a fixed budget. The channel-wide form is useful when the constraint is a shared resource — the consumers on that channel all write to the same database connection pool, so limiting the total in flight matters more than limiting per consumer. The practical relevance is that people set a prefetch expecting one behaviour and get the other, then are confused about memory usage or distribution. Note that RabbitMQ's interpretation of global differs from the AMQP specification, which is a known deviation and worth being aware of when reading the spec. And remember prefetch has no effect at all with automatic acknowledgement, since nothing is ever outstanding.

The first decision is whether to hold the message during the work or to acknowledge and track state elsewhere. Holding it — processing then acking — gives you redelivery on failure for free, which is the main benefit of a broker. The constraints are that the connection must stay alive and that a low prefetch is needed so one slow message does not block a batch. The risk is consumer timeout. RabbitMQ has a delivery acknowledgement timeout, 30 minutes by default in recent versions, after which the channel is closed and the message requeued. Work longer than that fails repeatedly with a confusing error. Raise the timeout or restructure. Restructuring usually means acknowledging quickly and recording the work in a database with its own status, so progress survives independently of the message. The broker is then used for triggering rather than for tracking. Other practical measures: keep prefetch at 1 for long tasks so distribution is fair, scale consumers horizontally rather than making one faster, and ensure the work is idempotent since a redelivery after partial completion is likely. And never hold a database transaction open across the whole task.

Any message it had received but not acknowledged is requeued and delivered to another consumer — or back to the same one when it reconnects. That is the core reliability guarantee of manual acknowledgement, and it is why unacked messages are tracked per channel: closing the channel or the connection releases them. The consequences to design for. The message will be processed again, possibly after being partially processed the first time. If the consumer had written half its changes before dying, the redelivery must cope with that — which is the idempotency requirement. The redelivered flag is set on the second delivery, so a consumer can tell it may be a repeat. It is a hint rather than a guarantee, since a message can be redelivered with the flag in situations you did not cause. With a high prefetch, a consumer death requeues everything it held, which can be a large batch arriving at once elsewhere. The operational detail worth knowing is detection time. If the process dies the TCP connection closes and requeue is immediate. If the machine vanishes without closing the socket, the broker waits for its heartbeat to fail — so heartbeat configuration determines how long messages are stuck.

Add more consumers on the same queue. The broker distributes messages among them, so throughput scales roughly linearly until something else becomes the bottleneck. What usually limits it, in order of likelihood. The downstream: consumers writing to a database will saturate its connection pool or its write capacity long before the broker struggles. Adding consumers then makes things worse by increasing contention. The queue itself: a single queue is handled by one Erlang process on one node, so a single queue has a throughput ceiling regardless of consumer count. Sharding across several queues is the answer when you hit it. Prefetch: too low and each consumer waits a round trip per message, capping throughput. Too high and distribution becomes uneven. The network and the broker's own resources, particularly with persistent messages where disk I/O is the constraint. The diagnostic order is to check whether the queue is growing while consumers are idle — which means the downstream is the limit — or whether consumers are busy and the queue is draining slowly, which points at consumer capacity or prefetch.

It allows several consumers to subscribe to a queue while only one receives messages at a time. If that consumer disconnects, another is promoted automatically. The purpose is ordered processing with high availability. Ordering requires a single consumer, but a single consumer is a single point of failure — if it dies, processing stops until someone restarts it. Single active consumer gives you the ordering guarantee of one consumer plus automatic failover. It is also useful when the work must not be done concurrently for correctness reasons — a consumer that maintains in-memory state, or one that would deadlock if two instances ran. The cost is that you get no parallelism, so throughput is bounded by one consumer regardless of how many are connected. The extra consumers are pure standby capacity. The usual alternative when you need both ordering and throughput is partitioning: route messages to several queues by a hash of the entity key, with one active consumer per queue. Order is preserved per entity while different entities process in parallel — which is almost always the ordering guarantee actually required, rather than global order. The consistent hash exchange plugin supports that routing.

Shut down gracefully: stop accepting new deliveries, finish what is in flight, acknowledge, then close the connection. Most clients support cancelling the consumer, which tells the broker to stop sending while leaving the connection open so outstanding messages can still be acknowledged. Killing the process instead requeues everything unacked, which is safe but causes duplicate processing and a burst of redelivery. The orchestrator must cooperate. Kubernetes sends SIGTERM then waits for the termination grace period before SIGKILL. If your grace period is 30 seconds and a message takes two minutes, you will be killed mid-processing on every deployment. Either raise the grace period or keep tasks short. A lower prefetch helps here too — a consumer holding one message drains in one message-time, while one holding a hundred takes far longer. During a rolling deployment old and new consumers coexist, so both must handle the same message format. That is the same backward-compatibility discipline as an API: additive changes only, consumers tolerant of unknown fields. And because redelivery is likely during any deployment, idempotency is not optional.

Losing a message is possible at every hop, so reliability requires settings at each one. On the publisher: publisher confirms enabled, so you know the broker accepted it, and the mandatory flag with a return handler so you know it was routable. Without confirms a publish is fire-and-forget. On the message: delivery_mode 2, making it persistent. On the queue: durable, so the queue survives a broker restart. Ideally a quorum queue, so it survives losing a node. On the consumer: manual acknowledgement, acking only after the work is genuinely complete. Auto-ack loses the message if the consumer dies mid-processing. On failure handling: a dead-letter exchange, so a rejected message goes somewhere inspectable rather than being discarded. And on the write side: the transactional outbox, so the message cannot be published without the state change committing, or vice versa. The cost of all this is throughput — every step adds a disk write or a round trip. So the honest framing is that you choose the level of guarantee per queue, and use the full set only where loss actually matters.

AMQP has a transaction mechanism — tx.select, tx.commit — that makes publishes atomic. It works, and it is extremely slow. The reason is that a transaction commit is synchronous and blocking: the publisher waits for the broker to complete the whole commit before continuing, and the broker must fsync. Throughput drops by an order of magnitude or more compared to unconfirmed publishing. Publisher confirms achieve the same reliability guarantee asynchronously. The publisher keeps sending, and the broker acknowledges each message by sequence number as it becomes safe. The publisher tracks outstanding sequence numbers and only worries about the ones not yet confirmed. That pipelining is the whole difference — you get the guarantee without serialising on each message. Confirms also handle the multiple flag, acknowledging everything up to a sequence number at once, which reduces the acknowledgement traffic further. The cost is client complexity: you must maintain the outstanding set and handle nacks by republishing. Batch confirms are a middle ground — publish a hundred, wait for the batch — which is simpler than fully asynchronous handling and much faster than transactions. RabbitMQ's own documentation recommends confirms over transactions for this reason.

A nack means the broker could not take responsibility for the message — typically an internal error, or the broker running out of resources. It is rare, which is precisely why it is usually unhandled and why the failure is silent when it happens. The correct response is to republish. That requires knowing which message the nack refers to, which is why you must retain the message keyed by its publish sequence number until it is confirmed. Without that bookkeeping you know something failed but not what. Republishing should back off, since a nack often means the broker is under pressure and immediate retry makes it worse. After a bounded number of attempts, the message must go somewhere durable — a local outbox table, a file, or an alert — rather than being dropped. Losing it silently at this point defeats the entire reliability chain. The related case is a returned message from the mandatory flag, which means routable-nowhere rather than broker failure. Retrying that is pointless; it needs logging and usually a topology fix. The practical point is that both handlers must exist. Enabling confirms without handling nacks gives you the illusion of reliability.

RabbitMQ enforces a maximum time a consumer may hold an unacknowledged message — 30 minutes by default since version 3.8.15. Exceed it and the channel is closed with a PRECONDITION_FAILED error, and the message is requeued. It exists to detect consumers that have silently stopped making progress. Without it, a hung consumer holds messages indefinitely and they are never redelivered — the queue appears to have consumers and nothing moves. The problem it causes is for legitimately long work. A consumer processing a large file for 45 minutes hits the timeout, its channel closes, the message is requeued, another consumer picks it up, and the cycle repeats forever while the work is never completed. The error message points at the channel rather than at the duration, so it is not obvious. The options: raise consumer_timeout in the broker configuration, which is a global setting rather than per-queue. Or restructure so the message is acknowledged quickly and progress is tracked in your own database, with the broker used only to trigger the work. The second is better architecture anyway, because it makes progress survivable independently of broker state.

Heartbeats are periodic frames exchanged between client and broker to confirm the connection is alive. If two consecutive intervals pass with nothing received, the peer is considered dead and the connection is closed. They matter because TCP does not reliably tell you a peer has vanished. If a machine loses power or a firewall silently drops the connection, both sides can hold a socket that will never carry data again — a half-open connection. Without heartbeats, the broker keeps that consumer's unacknowledged messages allocated to a consumer that no longer exists, and they are never redelivered. The default is 60 seconds, so detection takes up to about two intervals. Too long a timeout means slow failure detection and messages stuck. Too short means false positives — a consumer doing heavy synchronous work on the same thread as the connection may fail to send heartbeats and be disconnected despite being healthy. That is a real failure mode with single-threaded clients doing long processing. So the practical guidance is to keep the default unless you have a reason, and to ensure the client library sends heartbeats on a separate thread from message processing, which most do.

Reconnect automatically with backoff, and recover the topology. Most client libraries offer automatic recovery: on connection loss they reconnect, reopen channels, redeclare exchanges and queues that the client declared, restore bindings, and resume consumers. Enabling it is usually one flag and it handles the common case well. The backoff matters. Without it, a broker restart means every client reconnecting simultaneously, which is a thundering herd that can prevent the broker from coming up. Exponential backoff with jitter spreads them. What automatic recovery cannot do is reconstruct application state. Messages that were unacknowledged at the time of the failure are requeued and will be redelivered — possibly to this consumer, possibly elsewhere — so partial work must be idempotent. On the publisher side, messages published but not yet confirmed at the moment of disconnection are in an unknown state. They may or may not have reached the broker, so republishing risks duplication and not republishing risks loss. Since you cannot tell, republish and rely on consumer idempotency. And expose connection state as a health signal, so an application that cannot reach the broker fails its readiness check rather than silently doing nothing.

A message that always fails processing, and is requeued each time, loops forever. It is redelivered immediately, fails again, and spins at full speed. The effects are worse than a single stuck message. It consumes a consumer slot continuously, fills logs, burns CPU, and because it is always at the head of the queue it can block everything behind it. A single malformed message can effectively halt a pipeline. The cause is almost always rejecting with requeue true on a deterministic failure — a parse error, a schema mismatch, a null field. Those will never succeed no matter how many times you retry. The fixes. Distinguish transient from permanent failures in the consumer: retry the transient ones, reject the permanent ones without requeue so they dead-letter. Where you cannot tell, bound the retries. Track an attempt count in a header, increment on each redelivery, and dead-letter past a threshold. RabbitMQ does not count redeliveries for you on classic queues — the x-death header from dead-lettering is the usual mechanism, or quorum queues offer a delivery-limit setting that dead-letters automatically. And always configure a dead-letter exchange, so a poison message ends up somewhere inspectable.

RabbitMQ has no built-in delayed retry, so you build it from TTL and dead-lettering. The standard pattern: the main queue dead-letters to a retry exchange. The retry queue has a message TTL and dead-letters back to the main exchange. A failed message goes to the retry queue, waits for the TTL, expires, and is routed back for another attempt. For exponential backoff you need several retry queues with increasing TTLs — 10 seconds, 1 minute, 5 minutes — and the consumer routes to the appropriate one based on the attempt count in a header. A single retry queue with variable per-message TTL does not work, because head-of-line blocking means a message with a long TTL delays the ones behind it. The attempt count comes from the x-death header that dead-lettering adds, which records how many times a message was dead-lettered from each queue. After the final retry, route to a dead-letter queue for human inspection rather than retrying forever. The delayed message exchange plugin simplifies this considerably if you can install it, since it supports arbitrary per-message delays without the queue-per-delay structure.

When a message is dead-lettered, RabbitMQ adds an x-death header recording the history: which queue it came from, why it was dead-lettered — rejected, expired, or maxlen — the exchange and routing keys, a timestamp, and a count of how many times this has happened from that queue. That count is the practical value. It is the only built-in mechanism on classic queues for knowing how many times a message has been through a retry cycle, which is what you need to bound retries and eventually give up. So a retry consumer reads the count from x-death, decides whether to retry again or to route to the final dead-letter queue, and chooses which backoff tier to use. The reason field is also useful diagnostically: distinguishing a message that was actively rejected from one that expired or was dropped for queue length tells you very different things about what went wrong. The caveat is that the header is only added on dead-lettering, so a message requeued with basic.nack and requeue true carries no history at all — which is another reason to prefer dead-lettering over requeuing for failures. Quorum queues additionally offer a delivery count, which is simpler to use.

RabbitMQ applies flow control rather than crashing, which is better but still disruptive. When memory use crosses the high watermark — 40% of system memory by default — the broker blocks publishing connections. Publishers are not disconnected; they simply stop being able to send, and a publish call blocks. Consumers continue, so the queue drains and the broker eventually unblocks. The disk free space limit works similarly: below the threshold, publishers are blocked to prevent the disk filling entirely. The practical problem is that blocked publishers look like a hung application. A synchronous publish that never returns, with no error, is confusing to diagnose. Clients can register a blocked-connection listener to detect it, which most applications do not. The causes are usually a consumer outage letting a queue grow without bound, or unbounded queues generally. The preventions: set max-length or max-length-bytes policies on queues so they cannot grow indefinitely, use lazy or version-2 queues so large backlogs live on disk rather than in memory, alert on queue depth well before the watermark, and alert on the broker's memory alarm state directly. Bounded queues with a defined overflow behaviour are the real fix.

A policy setting max-length or max-length-bytes caps how much a queue holds. When full, the overflow behaviour decides what happens. drop-head discards the oldest message to make room for the new one. Appropriate for time-sensitive data where the newest is most valuable — live metrics, position updates. reject-publish refuses the new message and nacks the publisher, so the producer learns it could not be accepted and can decide what to do. This is genuine back-pressure and is usually the right choice for work that matters. reject-publish-dlx does the same but dead-letters the rejected message, so it is preserved for inspection. The important design point is that the default — unbounded — is a choice to fail catastrophically. An unbounded queue grows until the broker hits its memory alarm and blocks all publishers across the whole vhost, so one runaway queue takes down unrelated applications. Setting a limit converts that into a localised, predictable failure. The limit should be sized from how long you are willing to buffer during a consumer outage, not from a round number. And it needs a monitor, since a queue sitting at its limit is silently dropping or rejecting.

You do not achieve exactly-once delivery — you achieve at-least-once delivery plus idempotent processing, which is observationally equivalent. The reason exactly-once is impossible is that the acknowledgement can be lost. A consumer that processes a message and crashes before acking cannot distinguish that from having crashed before processing, and neither can the broker. Redelivery is the only safe choice. So the design is: at-least-once on the transport, deduplication in the consumer. The mechanism is a unique message identifier recorded atomically with the effect. Insert the ID into a processed-messages table in the same transaction as the business change; a duplicate hits the unique constraint and is skipped. Doing the check and the work in separate transactions reintroduces the race. Better still, make the operation naturally idempotent so no dedup table is needed — an upsert keyed by the entity, or setting a state rather than incrementing. For external side effects, push idempotency to that system with an idempotency key it honours. And retain deduplication keys long enough to cover your maximum redelivery window.

At-least-once is a transport guarantee: the message will be delivered, possibly more than once. It says nothing about what the consumer does. Effectively-once is an end-to-end property: despite duplicate delivery, the observable effect is as if the message were processed once. It is achieved by combining at-least-once transport with idempotent processing or transactional deduplication. The distinction matters because vendors market exactly-once and mean different things by it. Kafka's exactly-once semantics apply within Kafka — a read-process-write cycle entirely inside the cluster can be transactional. It does not extend to external side effects: if your consumer sends an email, Kafka cannot un-send it. So any effect outside the system providing the guarantee is your responsibility regardless of the transport. The practical framing for an interview: exactly-once delivery is not achievable across a network with independent failures, exactly-once processing is achievable within a transactional boundary, and effectively-once is what real systems build by making consumers idempotent. Saying a system provides exactly-once without qualifying the boundary is usually a sign of not having thought about the failure cases.

At three levels, because each catches different problems. Unit-test the handler as a plain function, separated from the broker. Given a deserialised message, does it produce the right effect? This should be the bulk of the tests and needs no infrastructure. Integration-test against a real broker with Testcontainers. This catches serialisation problems, routing key mistakes, binding errors, and acknowledgement handling — none of which a mocked broker would reveal, and all of which are common. An in-memory fake broker gives false confidence because routing semantics differ. Contract-test the message format so a publisher change that breaks consumers fails the build rather than production. Pact supports message contracts, and a schema registry enforces it structurally. The cases worth testing deliberately: a redelivered message, to verify idempotency; a poison message, to verify it dead-letters rather than looping; and a consumer that fails mid-processing, to verify requeue behaviour. Those failure paths are where the real bugs are, and they are almost never covered — most test suites only exercise the happy path, which is the part that rarely breaks.

A dead-letter exchange is where messages go when they cannot remain in their queue. It is configured on the queue with x-dead-letter-exchange. Three conditions trigger it. The message is rejected with basic.reject or basic.nack and requeue set to false. The message TTL expires. Or the queue exceeds its length limit and the overflow policy drops the message. Without a dead-letter exchange, all three cases discard the message silently. The value is that failures become visible and recoverable. A dead-letter queue you can inspect tells you what failed and why — the x-death header records the reason — and lets you fix the problem and replay the messages. The design points: give the dead-letter queue its own monitoring and alert on depth, because a dead-letter queue nobody looks at is the same as discarding. Set x-dead-letter-routing-key if you want to change the routing key on dead-lettering, otherwise the original is preserved. And be careful not to create a loop — a dead-letter queue that itself dead-letters back to the original exchange will cycle messages indefinitely, which is a real and easily-made configuration mistake.

Classify by whether the same input could succeed later. Transient failures are retryable: a database connection timeout, a downstream service returning 503, a lock contention failure, a network blip. Nothing about the message is wrong, so retrying has a genuine chance. Permanent failures are not: a malformed payload, a schema mismatch, a missing required field, a reference to an entity that does not and will never exist, a business rule violation. Retrying these produces identical failures forever. The consumer should distinguish them explicitly rather than catching everything and requeuing. That usually means catching specific exception types, or checking an HTTP status class for downstream calls. The ambiguous cases — a downstream returning 404, which might be a race or might be permanent — should be treated as retryable but with a bounded count, so a genuine permanent failure eventually dead-letters rather than looping. The rule to state clearly: unbounded retry is never correct. Even for transient failures, an extended outage means messages retrying continuously, and a bounded count with dead-lettering converts that into a manageable backlog you can replay once the dependency recovers.

Inspect, fix, and replay — deliberately, not automatically. Inspection means looking at why they failed. The x-death header gives the reason and the origin queue, and the payload tells you what the data was. Grouping by failure cause usually reveals that a large batch shares one root cause. Fixing may mean deploying a consumer fix, correcting reference data, or repairing the message itself if a publisher produced something malformed. Replaying means publishing them back to the original exchange. Tooling matters here: the shovel plugin can move messages between queues, and the management API can be scripted. Building a small replay tool is worth it if dead-lettering is at all common. What you should not do is replay blindly, since messages that dead-lettered for a permanent reason will simply return. The operational requirement is an alert on dead-letter queue depth. A dead-letter queue that fills silently over weeks is no better than discarding, and that is the usual outcome when nobody owns it. Also set a TTL or length limit on the dead-letter queue itself, or an unnoticed failure will eventually exhaust the broker.

The loop happens when a dead-letter path routes back to a queue that dead-letters again, with nothing incrementing or checking a counter. The classic case is a retry queue that dead-letters to the main exchange, whose queue dead-letters back to the retry queue. A permanently-failing message cycles forever, consuming resources and generating log noise indefinitely. The prevention is a bounded attempt count. Read the count from the x-death header, and once it exceeds a threshold route to a terminal dead-letter queue that has no dead-letter exchange configured. That queue is the end of the line. Quorum queues offer delivery-limit, which dead-letters automatically after a configured number of deliveries — simpler and less error-prone than header bookkeeping. The other prevention is topology discipline: draw the dead-letter graph and check it is acyclic. It is easy to create a cycle accidentally when several queues share a dead-letter exchange. And monitor for it: a message being redelivered thousands of times shows up as a high message rate on a queue with no useful throughput, which is a good alert condition even if you believe the topology is safe.

It depends on what you want to happen to it, and the choice must be deliberate. Acknowledging means the message is gone. That is correct only if you have durably recorded the failure somewhere else — written it to a failure table, logged it with enough detail to reconstruct, or published it to another queue yourself. Acking and only logging a stack trace means the data is lost. Rejecting without requeue is usually the right answer, because with a dead-letter exchange configured the message is preserved automatically with its failure metadata. Rejecting with requeue is right only for transient failures, and only with a bounded retry count. The anti-pattern is a catch-all that logs and acks. It makes the consumer look healthy — no errors surface to the broker, the queue drains, metrics look fine — while silently discarding data. That failure mode can run for months undetected. So the rule: never ack a message you did not process, unless you have explicitly persisted it elsewhere. And make the dead-letter path the default for anything you cannot handle, so the decision to discard is never implicit.

Design for it before it happens, because during any rolling deployment both old and new consumers run simultaneously against messages of both shapes. The discipline is the same as API compatibility: add optional fields, never remove or rename, never change a field's type or meaning. Consumers must ignore unknown fields, and that expectation has to be established from the first release. Include a schema version in the message headers so a consumer can branch if it must, and so you can tell from a dead-lettered message which shape it was. For a genuinely breaking change, the safe sequence is to publish both formats for a transition period, or to route the new format to a different routing key so new consumers bind to it while old ones continue on the old key. Deploy consumers first so they can handle both, then switch publishers, then retire the old path. A schema registry with compatibility checking — as used with Avro — enforces this structurally by rejecting an incompatible schema at publish time, which is much stronger than a convention. Messages already sitting in a queue during a deployment are the case people forget: they were published under the old schema and will be consumed by new code.

The shovel plugin moves messages from a source queue to a destination, which can be on the same broker or a different one, continuously and reliably. The uses are mostly operational. Replaying a dead-letter queue back to the main queue after fixing the underlying problem — this is the most common day-to-day use, and it beats writing a bespoke script. Migrating between brokers: point a shovel at the old broker's queues and drain them into the new one, which allows a cutover without losing what was in flight. Bridging environments or data centres, moving messages from a local broker to a central one. And draining a queue that must be emptied without a consumer. The difference from federation is intent. A shovel is a point-to-point transfer of a specific queue, configured on one side, and it consumes from the source so messages move. Federation links exchanges or queues across brokers as an ongoing topology, and is more about geographic distribution. Shovels can be configured dynamically through the management API, which makes them convenient for one-off operational tasks rather than requiring a config change and restart.

The most important metric is queue depth over time, per queue. A growing queue means consumption is not keeping up, and it is the leading indicator of almost every problem. Alongside it, the consumer count per queue — a queue with zero consumers is usually a deployment or crash, and it is the fastest explanation for a growing backlog. Message rates in and out, so you can see whether the imbalance is a publish spike or a consumption drop. Unacknowledged message count, which reveals consumers holding messages without progressing. Dead-letter queue depth, alerted on any non-zero value in most systems. Broker-level: memory and disk alarm state, since a triggered alarm blocks all publishers; file descriptor and socket usage; and Erlang process count. Connection and channel counts, since leaks are common and eventually exhaust the broker. And message age — the time the oldest message has been waiting — which is often more meaningful than depth, because a queue of 10,000 draining in seconds is fine while a queue of 50 sitting for an hour is not. The Prometheus plugin exposes all of this directly.

Establish first whether consumers are actually consuming. Check the consumer count on the queue — a healthy-looking process that failed to subscribe registers zero, which is common after a configuration change. Then check the unacknowledged count. If it is high and static, consumers have messages but are not completing them — they are stuck, usually blocked on a downstream call with no timeout. A thread dump on a consumer confirms it. If unacknowledged is near zero and the queue still grows, consumers are not being sent messages. Check whether prefetch is set unusually low, and whether the broker has a memory alarm active which throttles delivery. Compare the publish rate to the consume rate. If publishing has spiked, the problem is upstream and the queue is doing its job absorbing it — the question becomes whether it will drain in time. Check for a poison message: a very high redelivery rate with no throughput means one message is looping. And check the downstream the consumers write to. Most often the consumers are healthy and the database is the constraint, in which case adding consumers makes it worse.

Enough to reconstruct what happened to a specific message without logging the message content indiscriminately. On receipt: the message ID, the correlation ID, the queue, and the redelivered flag. The redelivered flag is particularly valuable and almost never logged — it tells you immediately whether a failure is a first attempt or a repeat. On completion: the outcome and the duration. On failure: the exception, the message ID, the attempt count, and the decision taken — retried, dead-lettered, or acked. Propagate the trace context from the message headers into your logging context, so the consumer's logs join up with the publisher's. Without that, a distributed flow is impossible to follow, and it is the single most valuable thing to get right. What to avoid: logging full payloads by default, which is a volume problem and often a personal-data problem. Log identifiers and log the payload only on failure, and even then consider redaction. And log at the right level — a retryable failure is a warning, a dead-lettered message is an error that someone should act on.

Propagate a trace context in the message headers, using the W3C traceparent format so standard tracing tools understand it. The publisher injects the current trace context when publishing. The consumer extracts it and continues the trace, so the consumer's span is a child of the publisher's. Any further publishing continues the chain. Without this, a distributed flow breaks at every queue — you have traces on either side with no link, and correlating them means matching timestamps and guessing. OpenTelemetry has messaging conventions defining the span structure and attributes, and most client libraries have instrumentation that does the injection and extraction automatically. The subtlety worth mentioning is that the publisher-consumer relationship is not a normal parent-child call, because it is asynchronous and one message may have many consumers. The convention uses span links rather than strict parenting for that reason, so the trace reflects the actual causality. Also carry a business correlation ID separately — an order ID or a saga ID — because trace IDs are per-request while a business flow may span many. Being able to find everything related to one order is what support actually needs.

Not setting messages persistent, or setting them persistent on a non-durable queue, and believing the data is safe. Using automatic acknowledgement because it is the default in some clients, which silently loses messages on consumer failure and disables prefetch. Leaving prefetch unlimited, so one consumer hoards messages and distribution is uneven. Requeuing on every failure, producing poison-message loops. No dead-letter exchange, so failures vanish. Unbounded queues, so one stuck consumer eventually triggers the broker memory alarm and blocks publishing for every application on the vhost. Sharing a channel across threads, which corrupts framing. Publishing without confirms and assuming the message arrived. Not handling unroutable messages, so a routing key typo discards silently. Assuming ordering that RabbitMQ does not provide. And the architectural one: using a broker for request-reply where the caller waits, which adds a component and latency for no decoupling benefit. The common thread is that most of these fail silently rather than loudly, which is why they survive into production — the system appears to work until the day it does not.

It depends on whether the messages still exist. If they dead-lettered, they are in the dead-letter queue and can be shovelled back to the original exchange once the fix is deployed. That is the straightforward case and is the main reason dead-lettering matters. If they were acknowledged and processed incorrectly, they are gone from the broker. RabbitMQ queues are destructive — once acked, there is no history. Recovery then depends on whether the source data still exists upstream, so you republish from the system of record. That asymmetry is worth stating in an interview, because it is a real architectural limitation compared to a log-based system. Kafka or RabbitMQ streams retain messages independently of consumption, so replay is a matter of resetting an offset. The practical mitigations if replay matters: use streams rather than queues for events you might need to reprocess; or archive every message to durable storage as it is published, which gives you a replay source at the cost of storage; or ensure the upstream system of record can regenerate the events. Deciding this before you need it is much cheaper than after.

Rejecting is a deliberate signal with a chosen outcome. Crashing is an accident with a default outcome. When you reject with requeue false and a dead-letter exchange is configured, the message is preserved with failure metadata, and the consumer continues with the next message. That is controlled. When a consumer crashes or its connection drops, every unacknowledged message it held is requeued — not just the one that caused the problem. With a high prefetch that could be dozens of messages, all redelivered, all reprocessed. The one that caused the crash is redelivered too, so if it is deterministic the new consumer crashes as well, and you have a crash loop that takes out consumers one after another. That cascade is the practical danger, and it is why an uncaught exception in a message handler is worse than an uncaught exception in a request handler. So consumers should catch broadly at the top of the handler, classify the failure, and reject or ack deliberately — letting nothing escape to kill the connection. The exception is genuinely unrecoverable process state, where crashing and being restarted is correct.

A cluster is several nodes sharing metadata — users, vhosts, exchanges, bindings and policies are replicated to every node. Queue contents are not replicated by default. A classic queue lives on one node, and other nodes route to it. So a cluster gives you a single logical broker and shared topology, but the loss of a node means its queues become unavailable. Replication of contents requires quorum queues or streams, which maintain copies across nodes using Raft. Clients can connect to any node and be routed transparently, which is what makes the cluster look like one broker. The important operational constraints. Nodes must be on a low-latency network — clustering is not designed for links across regions, and a slow link causes spurious partition detection. For geographic distribution you use federation or shovels between separate clusters instead. All nodes must run compatible versions and share an Erlang cookie for authentication. And cluster size should be odd for quorum queues, since Raft needs a majority: three nodes tolerate one failure, five tolerate two. An even-sized cluster gives no additional fault tolerance over the odd number below it.

A partition is when cluster nodes cannot reach each other but are all still running — a split brain, where each side may believe the other has failed. It is dangerous because both sides can continue accepting writes, and reconciling divergent state afterwards may be impossible without losing data. RabbitMQ offers several partition handling modes. ignore does nothing and leaves you with a split cluster, which is only safe if partitions genuinely cannot happen. pause_minority stops the nodes on the smaller side, so only the majority side continues serving — this preserves consistency and is the usual recommendation. autoheal picks a winning partition after the fact and restarts the losers, favouring availability over consistency. pause_minority is the CP choice and autoheal is the AP choice, which is a useful way to frame it. Quorum queues handle this more gracefully than classic mirrored queues did, because Raft only accepts writes with a majority, so a minority partition cannot accept conflicting writes in the first place. The practical prevention is to run the cluster on a reliable low-latency network, and to tune the net_ticktime so brief network hiccups are not misread as partitions.

The client is given a list of node addresses and tries them in turn, or connects through a load balancer fronting the cluster. Most client libraries accept multiple hosts and will attempt the next on connection failure, with automatic recovery reconnecting and restoring topology and consumers. A load balancer is simpler for clients but adds a component and can obscure which node you are actually on, which complicates debugging. It also needs a health check that reflects broker readiness rather than just TCP acceptance. The important caveat is that connecting to any node does not make queues available. With classic queues, a queue lives on one node — if that node is down, the queue is unavailable regardless of which node you connected to. Failover of the connection does not mean failover of the data. Quorum queues change that: the queue has replicas, and if the leader's node fails a follower is elected, so the queue remains available through the surviving nodes. So client failover and queue availability are separate concerns, and getting the first right without the second gives you a cluster that reconnects successfully to an unusable queue.

Federation links brokers that are not clustered, forwarding messages between exchanges or queues across the link. The key difference is coupling. A cluster is a single logical broker with shared metadata, requiring a fast reliable network and matching versions. Federated brokers are independent — separate clusters, separate metadata, possibly different versions — connected only by the federation link. That makes federation the tool for wide-area links. Two data centres, or a regional broker forwarding to a central one, where clustering would be fragile because of latency and would risk partitions. Federation is also tolerant of the link going down: it reconnects and resumes, whereas a cluster treats an interruption as a partition. A federated exchange forwards messages published to an upstream exchange into a downstream one. A federated queue moves messages from an upstream queue when the downstream has consumers that need work — useful for spreading load geographically. The difference from a shovel is scope and intent: a shovel is a point-to-point drain of a specific queue, often temporary and operational, while federation is ongoing topology linking. Both are plugins and both are configured dynamically.

Rolling upgrade, one node at a time, provided the versions are compatible for mixed operation — which RabbitMQ supports within defined version ranges but not across all jumps, so the release notes matter. The sequence per node: stop the node, upgrade, restart, wait for it to rejoin and for quorum queues to resynchronise, then move to the next. Never take down two nodes of a three-node cluster simultaneously, or you lose quorum and writes stop. With quorum queues, the leader migrates when its node stops, so the queue stays available. With classic non-mirrored queues, the queues on the stopped node are simply unavailable for the duration — which is why a cluster of classic queues cannot be upgraded without impact. Clients must handle reconnection, which is another reason automatic recovery and multiple host addresses matter. For a major version jump where mixed-version operation is not supported, the options are a full cluster stop, or standing up a new cluster and migrating with a shovel and a controlled cutover — the blue-green approach, which is safer but requires the topology and the clients to be redirected. Always test the upgrade path on a copy first.

Three nodes is the usual starting point, because quorum queues need a majority and three is the smallest cluster tolerating one failure. Going to five tolerates two failures but increases replication cost, since every message is written to more nodes. Beyond five, the consensus overhead outweighs the availability gain for most workloads. More nodes do not directly increase throughput for a single queue, because a queue is served by one leader process. Scaling throughput means more queues — sharding by key across several queues — rather than more nodes. Sizing each node comes down to memory, since the broker holds message backlogs and connection state; disk throughput, since persistent messages are fsynced; and file descriptors, since each connection and each queue consumes them. The usual guidance is to keep the memory high watermark well above your expected peak backlog, and to use fast disks for persistent workloads — disk latency is frequently the real throughput limit. The most important sizing input is the backlog you must survive: how long a consumer outage you want to absorb, multiplied by the publish rate and the message size. That number drives memory and disk far more than throughput does.

The queue becomes unavailable. Its messages are inaccessible until the node returns, and if the queue was durable the messages are recovered when it does — if it was not durable, they are gone. Clients connected to other nodes can still connect, but operations against that queue fail. Publishers routing to it get errors or, depending on configuration, silently unroutable messages. This is the fundamental limitation of classic queues in a cluster: clustering replicates metadata, not queue contents. The cluster survives; the queue does not. The historical answer was mirrored queues, which replicated contents but had correctness problems under partition and are now removed. The current answer is quorum queues. With replicas across nodes and Raft consensus, losing the leader's node triggers an election and a follower takes over, so the queue remains available and no acknowledged message is lost. The cost is more disk, more network traffic, and lower peak throughput than a single-node classic queue. The practical guidance is to use quorum queues for anything whose availability matters, and classic queues only for transient data where losing the queue is acceptable.

Separate the definitions from the messages, because they are backed up differently and matter differently. Definitions — vhosts, users, permissions, exchanges, queues, bindings, policies — can be exported as JSON through the management API or UI, and imported into a new broker. This is the important backup: it recreates your topology, and it should be in version control rather than only in a backup, so topology changes are reviewed. Messages are much harder. The on-disk message store can be copied only with the node stopped, and restoring it requires the same version and node name. It is not a practical routine backup mechanism. The realistic position is that a message broker is not a database and should not be treated as a system of record. Messages in flight are transient by design. If losing the current queue contents would be unacceptable, the fix is architectural — a transactional outbox so the source of truth is your database and messages can be republished, or streams with retention so history exists. So the answer is: version-control your definitions, and design so that message loss is recoverable from upstream rather than requiring a broker restore.

Start with the defaults, because they are the usual finding. The guest user exists with a known password and is restricted to localhost — delete it rather than relying on that restriction. Enable TLS for client connections and for inter-node communication. Without it, credentials and message contents cross the network in plaintext. Use per-application users with least-privilege permissions. RabbitMQ permissions are per vhost and per operation — configure, write and read — expressed as regular expressions over resource names. An application that only publishes should not have read permission on queues. Separate applications into vhosts so a compromised credential is scoped. Do not expose the management UI publicly. It is a full administrative interface, and it is a common accidental exposure. Use a proper authentication backend where available — LDAP or OAuth 2 via plugins — rather than managing internal users. And remember that authorisation is per resource name, so a naming convention that groups an application's resources by prefix makes the permission regexes tractable. Without one, permissions end up over-broad because the alternative is unmanageable.

The management plugin provides an HTTP API and a web UI for administering the broker — viewing queues, publishing test messages, inspecting bindings, managing users and policies, and exposing metrics. It is genuinely valuable for operations and for debugging routing, since it can trace how a routing key would be handled. What you should not do with it is treat its HTTP API as a hot path. Polling queue statistics frequently, particularly on a broker with many queues, is expensive — the statistics gathering competes with message handling and has been a real cause of broker degradation. Use the Prometheus plugin for metrics instead, which is designed for scraping. Do not use it to consume messages in production. The UI's get-message function acknowledges or requeues in ways that are easy to get wrong, and requeuing changes message order. Do not expose it to the internet, since it is a full administrative surface. And do not rely on it for automation — the definitions export is useful, but topology should be managed as code rather than clicked into the UI, or environments drift and nobody knows what the intended state is.

Several things, in roughly this order of likelihood. Persistence. Writing every message to disk and fsyncing is the dominant cost for durable workloads. Transient messages in memory are an order of magnitude faster. Single-queue concurrency. A queue is handled by one Erlang process, so a single queue has a throughput ceiling regardless of cluster size or consumer count. Sharding across queues is the fix. Replication. Quorum queues write to a majority of nodes, so throughput is bounded by the slowest replica and the network between them. Small messages with per-message acknowledgement, where the round trips dominate. Raising prefetch and using multiple-ack amortises this. Connection and channel churn. Opening a connection per message is catastrophically slow, and it happens more often than you would expect. The consumer's downstream, which is very often the real limit — the broker is idle while consumers wait on a database. The diagnostic order is to check whether the broker is actually saturated before tuning it. Most reported RabbitMQ throughput problems turn out to be consumer or downstream problems, and the broker was never the constraint.

The first thing to understand is that adding consumers to one queue helps only until the queue's own single-process ceiling is reached. Beyond that, the queue is the bottleneck and more consumers do nothing. The answer is sharding: split the logical stream across several queues and consume from all of them. The consistent hash exchange plugin routes by a hash of the routing key, so messages for the same key always land on the same queue. That preserves per-key ordering while parallelising across keys, which is usually the ordering guarantee you actually need. Alternatively the sharding plugin creates and manages a set of queues automatically. Other measures: raise prefetch so consumers are not round-tripping per message, batch acknowledgements with the multiple flag, and use transient messages if durability is not required for that stream. For genuinely very high volume, streams are the better primitive than queues, since the append-only design avoids the per-message bookkeeping. And check whether you need the volume at all — batching several logical events into one message often gives a bigger win than any tuning, because the per-message overhead dominates for small payloads.

Batching helps when per-message overhead dominates, which it does for small messages. There are two kinds. Batching at the application level means putting several logical events into one message. That reduces broker work proportionally and is usually the biggest win, but it couples the events — a failure means reprocessing the whole batch, and partial success needs handling. Batching at the protocol level means publishing many messages without waiting for each confirm, then waiting for a batch of confirms. That keeps messages independent while removing the round-trip serialisation, and it is almost always worth doing. On the consumer side, acknowledging with the multiple flag acknowledges everything up to a delivery tag in one frame, which reduces acknowledgement traffic considerably at high rates. The trade-offs to state: batching increases latency, because you wait to accumulate. It increases the blast radius of a failure. And it complicates idempotency, since a redelivered batch may be partially processed. So the guidance is protocol-level batching by default, and application-level batching only where throughput genuinely requires it and the events are naturally related.

Persistence means writing the message to disk and, for the guarantee to hold, fsyncing before confirming. That is typically an order of magnitude slower than in-memory handling, and disk latency becomes the throughput ceiling. RabbitMQ mitigates it by batching writes across messages, so high-throughput persistent publishing amortises better than the per-message cost suggests. But it remains the dominant expense for durable workloads. You can avoid it when losing messages on a broker restart is genuinely acceptable. Metrics, logs, cache invalidation notices, presence updates, and anything superseded by the next message a second later all qualify — replaying is unnecessary because the next value arrives shortly. You cannot avoid it for anything representing work that must happen or a fact that must be recorded. The useful framing is to decide per queue rather than globally. A system typically has a small number of queues carrying important work and a larger number carrying disposable signals, and treating them identically means either paying too much or risking too much. And remember persistence alone is insufficient — it needs a durable queue and publisher confirms to actually guarantee anything.

Connections are expensive: a TCP handshake, TLS negotiation if enabled, authentication, and per-connection memory and file descriptors on the broker. Opening one per message or per request is catastrophically slow and is a genuinely common mistake in code that treats the broker like an HTTP endpoint. The correct model is long-lived connections, created at startup and reused, with channels multiplexed over them. Channels are cheap but not free. Creating a channel per message adds broker work; pooling them per thread is the usual pattern. The threading rule matters more than the performance: a channel is not thread-safe, so sharing one across threads corrupts the protocol framing and produces errors that look like broker problems. At the other extreme, very large numbers of channels on one connection contend on the single TCP stream, so extremely high throughput may benefit from several connections. The operational symptom of getting this wrong is a broker with tens of thousands of connections, high memory use, and file descriptor exhaustion — usually caused by an application creating connections in a request handler and relying on garbage collection to close them.

Large messages are expensive: more network, more memory while queued, more disk if persistent, and more time to serialise and deserialise. A queue holding a backlog of large messages consumes memory proportionally, which is a common route to the broker's memory alarm. The usual guidance is to keep messages small and to pass references rather than payloads for anything large. Publishing an identifier and having the consumer fetch the data from a store or database keeps the broker lightweight, at the cost of an extra read and a consistency question — the data may have changed by the time the consumer reads it. That trade-off is worth naming: a self-contained message is a snapshot of the truth at publish time, while a reference is a pointer to current truth. Which you want depends on whether the consumer should act on what was true then or what is true now. For genuinely large payloads — files, images — the claim-check pattern is standard: store the object, publish only its location. And compress where payloads are large and compressible, though this shifts cost to CPU on both ends and is rarely worth it for small JSON.

A short queue is fast: messages arrive and are consumed while still in memory, with minimal bookkeeping. A long queue is slower for several reasons. Memory pressure builds until the broker pages messages to disk, and in older versions that paging happened in bursts that blocked the queue — a latency spike exactly when the system is already struggling. Version-2 classic queues and lazy queues avoid the cliff by writing to disk continuously, trading steady-state throughput for predictability. A long queue also means messages sit for a long time, so the meaningful metric becomes message age rather than depth. A queue of ten thousand draining in seconds is healthy; a queue of a hundred sitting for an hour is not. And a very long queue delays recovery: even after consumers are restored, working through the backlog takes time, and during that time latency for new messages is dreadful because they queue behind the backlog. The design conclusion is that queues should normally be near empty. A persistently non-empty queue means consumption is undersized, and the queue is masking it rather than solving it. Bound queue length so the failure is explicit.

Test the whole path, not the broker in isolation, because the broker is rarely the bottleneck and a broker benchmark tells you little about your system. Use realistic message sizes and realistic payloads, since throughput varies enormously with size and serialisation cost. Use the same durability settings as production. A benchmark with transient messages and auto-ack will show numbers you can never achieve with persistence and manual acknowledgement, and the difference is an order of magnitude. Test the failure paths deliberately: kill a consumer mid-processing and observe redelivery and duplicate handling; stop the downstream database and observe whether the queue bounds correctly or grows without limit; fill the queue past its limit and confirm the overflow behaviour is what you expected. Measure end-to-end latency — publish to processed — not just broker throughput, since that is what users experience. And test recovery: how long does it take to drain a backlog representing an hour of downtime? That number determines whether an outage is a blip or an extended degradation, and it is almost never measured until it matters. PerfTest is RabbitMQ's own tool for the broker-level portion.

Very few, because most RabbitMQ performance problems are application-side rather than broker configuration. The settings worth reviewing: the memory high watermark, which defaults to 40% of system memory. Raising it on a dedicated broker gives more headroom before publishers are blocked, though it is not a fix for an unbounded queue. The disk free limit, so the broker blocks publishing before the disk actually fills. consumer_timeout, if you have legitimately long-running consumers hitting the default 30 minutes. File descriptor limits at the OS level, since the default is often far too low for a broker with many connections and is a common cause of failures under load. net_ticktime, if brief network hiccups are being misinterpreted as partitions. What matters much more than any of these: queue length policies so queues are bounded, quorum queues where availability matters, prefetch on consumers, and connection reuse in clients. So the honest answer is that the first thing to tune is the application, and the broker configuration is mostly about setting safety limits rather than extracting performance.

More consumers when the bottleneck is consumer processing capacity and the queue itself is keeping up. This is the common case and it scales linearly until something else binds. More queues when the queue is the bottleneck. A single queue is served by one Erlang process, so there is a ceiling on messages per second through it regardless of how many consumers subscribe. When you hit that, adding consumers does nothing and you must shard. The diagnostic is to look at whether consumers are idle while the queue is backed up. Idle consumers with a growing queue means the queue is not delivering fast enough — that is the sharding signal. Busy consumers with a growing queue means you need more consumers, or the downstream is the limit. More queues also help when different message types have different processing characteristics: a slow expensive job type on the same queue as fast ones causes head-of-line blocking, and separating them lets each scale independently and gives you separate monitoring. That separation-by-workload argument is often more valuable than the throughput one — it stops one slow job type degrading everything else, which is a resilience benefit rather than a performance one.

Instead of putting a large payload in the message, store it somewhere durable — object storage, a database — and publish only a reference to it. The consumer uses the reference to fetch the data. The motivation is that brokers are poor at handling large payloads. Large messages consume broker memory while queued, slow persistence, increase network cost, and a backlog of them is a fast route to the memory alarm. Many brokers also have a hard message size limit. With the claim-check, the broker carries a few hundred bytes regardless of whether the payload is a kilobyte or a gigabyte. The considerations: the consumer needs access to the store, which adds a dependency. The stored object needs a lifecycle — deleting it after processing, or expiring it — or storage grows without limit. And you have introduced a second failure mode, where the message arrives but the object is missing. The subtle one is consistency: the object might be modified between publish and consume, so if the message should represent a snapshot the object must be immutable — write once with a unique key rather than overwriting. It is the standard approach for file processing pipelines.

When the caller needs the answer to proceed. A synchronous request-reply over a broker adds a component, adds latency, and gives you none of the decoupling — the caller is still blocked on the callee. An HTTP or gRPC call is simpler and easier to debug. When you need strong ordering across everything, since brokers give ordering only within narrow constraints and enforcing it costs all your parallelism. When the volume is trivial. A broker is a component to deploy, monitor, secure and upgrade. For a handful of events per hour, a database table polled by a job is less machinery and easier to reason about. When you need replay and history and you are reaching for a queue. Queues are destructive; a log is the right shape. And when the team has no operational capacity for it. An unmonitored broker with unbounded queues is a latent outage. The honest framing is that a broker buys decoupling in time, load and identity. If you are not getting at least one of those, you have added infrastructure for nothing — and the most common instance is using one for what is really a synchronous call.

Several consumers read from one queue, each message goes to exactly one of them, and throughput scales with consumer count. It is the standard pattern for distributing work. It does not apply when every consumer needs every message. Putting two services that both need order events on the same queue means each event goes to one of them, so half the work is silently skipped. That is a very common early mistake, and the symptom is confusing — both services appear to work, each just misses half the data. The fix is a queue per consumer, bound to the same exchange, so each gets its own copy. It also does not apply cleanly when ordering matters, since concurrent consumers finish in arbitrary order. Single active consumer or partitioning by key are the alternatives. And it interacts badly with long-running heterogeneous work: a consumer holding a slow message while others idle is a distribution problem, addressed by lowering prefetch or by separating slow work onto its own queue. The mental model worth stating: one queue means shared work, multiple queues mean broadcast. Choosing between them is choosing between distribution and duplication.

A saga breaks a distributed transaction into a sequence of local transactions, each with a compensating action that undoes it if a later step fails. There are two styles. Choreography: each service listens for events and reacts, publishing its own events. There is no coordinator, which keeps services decoupled, but the overall flow exists nowhere explicitly — understanding it means reading every service, and cycles are easy to create accidentally. Orchestration: a coordinator service drives the sequence, sending commands and reacting to replies. The flow is explicit and easier to reason about and monitor, at the cost of a component that knows about everyone. For anything beyond three or four steps, orchestration is usually worth the coupling. The hard parts are the compensations. They are not rollbacks — you cannot un-charge a card, you issue a refund, which is a different business event with its own visibility. Some actions cannot be compensated at all, so they should be ordered last. And the saga state must be persisted, so a crashed orchestrator resumes rather than abandoning half-completed work. Every step must be idempotent, since retries are certain.

Choreography means each service reacts to events and publishes its own, with no central controller. Orchestration means a coordinator explicitly directs the sequence. Choreography gives loose coupling: adding a service that reacts to an existing event requires no change anywhere. It suits simple flows and genuine event-driven design. Its weakness is visibility. The overall process is emergent — no single place describes it, so understanding what happens after an order is placed means tracing through several services. Debugging a stuck flow is hard because there is no state machine to inspect. And accidental cycles are easy to create, where service A reacts to B which reacts back to A. Orchestration makes the flow explicit and inspectable. You can query the saga state, see which step failed, and retry from there. Monitoring is straightforward. Its weakness is that the orchestrator knows about every participant, which is coupling, and it can grow into a god service holding all the business logic. The practical rule: choreography for simple reactive flows of two or three steps, orchestration once there is a genuine sequence with failure handling and compensations that someone needs to reason about.

Command Query Responsibility Segregation separates the write model from the read model, so each can be optimised independently. The write side handles commands and enforces invariants, typically with a normalised model. The read side serves queries from a shape built for reading — denormalised, pre-joined, possibly in a different store entirely such as Elasticsearch for search. Messaging is what connects them. The write side publishes events; a projector consumes them and updates the read model. The benefit is that queries stop constraining the write model. A reporting query that would require expensive joins against the transactional schema instead reads a purpose-built table. The cost is eventual consistency. A user who writes and immediately reads may not see their own change, which is confusing and must be designed for — either by reading from the write model for that case, or by making the UI optimistic. It also doubles the storage and adds a projector to operate and to rebuild when it goes wrong. So it is justified when read and write loads genuinely differ in shape or scale, and it is over-engineering for a typical CRUD service.

Event notification publishes a minimal event — something happened, here is the identifier. Consumers that need details call back to the source to fetch them. Event-carried state transfer publishes the relevant state within the event, so consumers need no callback. Notification keeps events small and always current, since the consumer fetches the latest data. But it creates a runtime dependency: the publisher must be available when consumers process, which undermines the decoupling the broker provided. It also produces a load spike on the publisher when many consumers react to one event. State transfer removes that dependency entirely — a consumer can process even if the publisher is down, which is the stronger form of decoupling. The costs are larger messages, duplicated data across services, and staleness: the event is a snapshot, so a consumer acting on it may be acting on out-of-date information. It also couples consumers to the event's schema more tightly, since they now depend on its fields. The practical choice depends on whether consumers need current truth or the truth at that moment. For audit and history, the snapshot is correct. For acting on current state, notification plus a fetch may be right.

Bound the call, isolate it, and decide the failure behaviour deliberately. Always set a timeout. A call with no timeout holds the message, holds the acknowledgement, and eventually hits the broker's consumer timeout — producing a requeue loop that looks like a broker problem. Add a circuit breaker. When the external service is comprehensively down, retrying every message wastes capacity and delays recovery. The breaker fails fast, and the consumer can then reject messages for retry rather than waiting. Decide what a failure means. If the call is essential, reject with a bounded retry so messages wait and are reprocessed when the service recovers. If it is optional, log and continue. Be careful about stopping consumption entirely during an outage: if you keep rejecting and retrying, you burn CPU. Pausing the consumer when the breaker is open, and resuming when it closes, is cleaner — the queue simply grows, which is what a queue is for. And ensure the external call is idempotent from your side, with an idempotency key, since redelivery after a timeout means you cannot tell whether the first call took effect.

SQS is a managed queue with a much simpler model: no exchanges, no routing, no AMQP. You put messages on a queue and take them off. Routing is done with SNS in front for fan-out. The advantages of SQS are operational: no cluster to run, effectively unlimited scale, and no capacity planning. The disadvantages are less flexibility — no complex routing, no priorities, limited message size, and at-least-once with best-effort ordering unless you use FIFO queues, which have lower throughput. SQS also uses visibility timeouts rather than persistent connections: a consumer polls, gets a message that becomes invisible for a period, and deletes it when done. If it does not, the message reappears. That is a different model from RabbitMQ's connection-bound acknowledgement, and it handles consumer death without heartbeats. RabbitMQ gives richer routing, lower latency, and full control, at the cost of operating it. The honest recommendation: if you are on a cloud and your routing needs are simple, a managed queue removes real operational burden. Choose RabbitMQ when you need its routing model, when you are not on a cloud, or when latency requirements are tight.

When the volume is low, when you already have the database, and when you want transactional atomicity with your business data. That last point is the strongest argument. Writing the work item in the same transaction as the state change gives you exactly the atomicity the outbox pattern reconstructs with a broker. There is no dual-write problem because there is only one write. It is also far less machinery: no broker to deploy, secure, monitor and upgrade, and the work items are queryable with SQL, which makes operational inspection trivial. The implementation needs care — SELECT FOR UPDATE SKIP LOCKED is the standard way to let several workers claim rows without contention, and it is well supported in PostgreSQL and MySQL. The limits are real. Polling adds latency and load. Throughput is bounded well below a broker. Fan-out to multiple consumers means implementing it yourself. And a heavily-used queue table generates significant write and vacuum pressure. So the rule of thumb: below a few hundred messages per second, with simple consumption, a table is often the better engineering choice. Above that, or with real routing needs, use a broker.

Incrementally, running both, because a cutover is high risk and rarely necessary. The usual approach is dual publishing: publishers write to both systems for a period. New consumers are built against the target, old consumers continue on the source, and you migrate consumers one at a time. Once all consumers are moved, publishing to the old system stops. That requires consumers to be idempotent, since during the overlap a message may be processed from both — or you route each consumer to exactly one source, which is cleaner. A bridge is the alternative: a process consuming from one and publishing to the other, so you migrate producers and consumers independently. Kafka Connect has connectors for this, and RabbitMQ shovels can feed a bridge. The genuinely hard part is not the transport, it is the semantic differences. Moving from Kafka to RabbitMQ means losing replay, so anything depending on it needs redesign. Moving the other way means losing per-message acknowledgement and redelivery, and gaining partition-based ordering — consumer error handling has to be rewritten, not ported. So scope the semantic gap before planning the mechanics.

Work along the path and find the first point where the message is not guaranteed. Is the publisher using confirms? Without them a publish is fire-and-forget and the message may never have reached the broker. Check whether nacks are being handled or silently ignored. Is the message routable? An unroutable message is confirmed and then discarded. Check for a mandatory flag with a return handler, or an alternate exchange, and look at the unroutable queue if one exists. A routing key typo is the single most common cause. Is the queue durable and the message persistent? Both are required; either alone loses data on restart. Is the consumer using automatic acknowledgement? If so, a crash mid-processing loses the message with no trace. Is there a dead-letter exchange? Without one, rejected and expired messages vanish. Does the queue have a length limit with drop-head overflow? That discards silently by design. And is the consumer acking on failure — a catch-all that logs and acks looks healthy while discarding data. The pattern is that almost every loss mechanism in RabbitMQ is silent by default, which is why the answer is to audit the configuration rather than to look for errors.