Where’s my message? Durability and you

There’s a dirty secret about creating queues and exchanges in Rabbit: by default they don’t survive reboot. That’s right; restart your RabbitMQ server and watch those queues and exchanges go poof (along with the messages inside).

The reason is because of a property on every queue and exchange called durable. It defaults to false, and tells RabbitMQ whether the queue (or exchange)
should be re-created after a crash or restart of Rabbit. Set it to true and you won’t have to re-create those queues and exchanges when the power supply in your server dies. You might
also think that setting durable to true on the exchanges and queues is all you need to do to make your messages survive a reboot, but you’d be wrong.
Whereas queues and exchanges must be durable to allow messages to survive reboot, it isn’t enough on its own.

A message that can survive a crash of the AMQP broker is called persistent. You flag a message as persistent by setting the delivery mode option of the message to 2 (your AMQP client may use a human-friendly constant instead)
before publishing it. At this point, the message is indicated as persistent, but it must be published to an exchange that’s durable and arrive in a queue that’s durable to survive. If
this weren’t the case, the queue (or exchange) a persistent message was sitting in when Rabbit crashed wouldn’t exist when Rabbit restarted, thereby orphaning the message. So, for a message
that’s in flight inside Rabbit to survive a crash, the message must

Have its delivery mode option set to 2 (persistent)

 Be published into a durable exchange

 Arrive in a durable queue

Do these three things and you won't have to play Where’s Waldo with your critical

messages.

The way that RabbitMQ ensures persistent messages survive a restart is by writing them to the disk inside of a persistency log file. When you publish a persistent message to a durable exchange, Rabbit won’t send the response until the message is committed
to the log file. Keep in mind, though, that if it gets routed to a non durable queue after that, it’s automatically removed from the persistency log and won’t survive a restart. When you use persistent messages it’s crucial that you make sure all three elements
required for a message to persist are in place (we can’t stress this enough).

Once you consume a persistent message from a durable queue (and acknowledge it), RabbitMQ flags it in the persistency log for garbage collection. If Rabbit restarts any-time before you consume a persistent message, it’ll automatically re-create theexchanges
and queues (and bindings) and replay any messages in the persistency log

into the appropriate queues or exchanges (depending on where in the routing process the messages were when Rabbit died).

You might be thinking that you should use persistent messaging for all of your messages. You could do that, but you’d pay a price for ensuring your messages survive Rabbit restarts: performance. The act of writing messages to
disk is much slower than just storing them in RAM, and will significantly decrease the number of messages per second your RabbitMQ server can process. It’s not uncommon to see a 10x or more decrease in message throughput when using persistency.

There’s also the issue thatpersistent messages don’t play well with RabbitMQ’s built-in clustering. Though

RabbitMQ clustering allows you to talk to any queue present in the cluster from any

node, those queues are actually evenly distributed among the nodes without redun-

dancy (there’s no backup copy of any queue on a second node in the cluster). If the

cluster node hosting your seed_bin queue crashes, the queue disappears from the

cluster until the node is restored … if the queue was durable. More important, while

the node is down its queues aren’t available and the durable ones can’t be re-created.

This can lead to black-holing of messages. We’ll cover the behavior in more detail and

show alternate clustering approaches to get around this in chapter 5.

Given the trade-offs, when should you use persistent/durable messaging? First, you

need to analyze (and test) your performance needs. Do you need to process 100,000

messages per second on a single Rabbit server? If so, you should probably look at

other ways of ensuring message delivery (or get a very fast storage system). For exam-

ple, your producer could listen to a reply queue on a separate channel. Every time it

publishes a message, it includes the name of the reply queue so that the consumer can

send a reply back to confirm receipt. If a message isn’t replied to within a reasonable

amount of time, the producer can republish the message. That said, the critical

nature of messages requiring guaranteed delivery generally means they’re lower in

volume than other types of messages (such as logging messages). So if persistent mes-

saging meets your performance needs, it’s an excellent way to help ensure delivery.

We use it a lot for critical messages. We’re just selective about what types of content

use persistent messaging. For example, we run two types of Rabbit clusters: traditional

RabbitMQ clustering for nonpersistent messaging, and pairs of active/hot-standby

nonclustered Rabbit servers for persistent messaging (using load balancers). This

ensures the processing load for persistent messaging doesn’t slow down nonpersistent

messages. It also means Rabbit’s built-in clustering won’t black-hole persistent mes-

sages when a node dies. Do keep in mind that while Rabbit can help ensure delivery, it

can never absolutely guarantee it. Hard drive corruption, buggy behavior by a con-

sumer, or other extreme events can trash/black-hole persistent messages. It’s ulti-

mately up to you to ensure your messages arrive where they need to go, and persistent

messaging is a great tool to help you get there.

A concept that’s related to the durability of a message is the AMQP transaction. So

far we’ve talked about marking messages, queues, and exchanges as durable. That’s all

well and good for keeping a message safe once RabbitMQ has it in its custody, but

since a publish operation returns no response to the producer, how do you know if

the broker has persisted the durable message to disk? Should the broker die before it

can write the message to disk, the message would be lost and you wouldn’t know.

That’s where transactions come in. When you absolutely need to be sure the broker

has the message in custody (and has routed the message to all matching subscribed

queues) before you move on to another t

ask, you need to wrap it in a transaction. If you come from a database background, it’s important
not to confuse AMQP transactions with what “transaction” means in most databases. In AMQP,
after you place a channel into transaction mode, you send it the publish you want to confirm, followed

by zero or more other AMQP commands that should be executed or ignored depend-

ing on whether the initial publish succeeded. Once you’ve sent all of the commands,

you commit the transaction. If the transaction’s initial publish succeeds, then the chan-

nel will complete the other AMQP commands in the transaction. If the publish fails,

none of the other AMQP commands will be executed. Transactions close the “last

mile” gap between producers publishing messages and RabbitMQ committing them

to disk, but there’s a better way to close that gap.

Though transactions are a part of the formal AMQP 0-9-1 specification, they have

an Achilles heel in that they’re huge drains on Rabbit performance. Not only can

using transactions drop your message throughput by a factor of 2–10x, but they also

make your producer app synchronous, which is one of the things you’re trying to get rid of with messaging. Knowing all of this, the guys at RabbitMQ decided to come up with a better way
to ensure message delivery: publisher confirms.

2

Similar to transactions,

you have to tell Rabbit to place the channel into confirm mode, and you can’t turn it

off without re-creating the channel. Once a channel is in confirm mode, every mes-

sage published on the channel will be assigned a unique ID number (starting at 1).

Once the message has been delivered to all queues that have bindings matching the

message’s routing key, the channel will issue a publisher confirm to the producer app

(containing the message’s unique ID). This lets the producer know the message has

been safely queued at all of its destinations. If the message and the queues are dura-

ble, the confirm is issued only after the queues have written the message to disk. The

major benefit of publisher confirms is that they’re asynchronous. Once a message has

been published, the producer app can go on to the next message while waiting for the

confirm. When the confirm for that message is finally received, a callback function in

the producer app will be fired so it can wake up and handle the confirmation. If an

internal error occurs inside Rabbit that causes a message to be lost, Rabbit will send a

message nack (not acknowledged) that’s like a publisher confirm (it has the message’s

unique ID) but indicates the message was lost. Also, since there’s no concept of mes-

sage rollback (as with transactions), publisher confirms are much lighter weight and

have an almost negligible performance hit on the Rabbit broker.

Now you have the individual parts of RabbitMQ down, from consumers and pro-

ducers to durable messaging, but how do they all fit together? What does the lifecycle

of an actual message look like from beginning to end? The best way to answer that is

to look at the life of a message in code.