Messaging: handing over work without holding the line

The previous lesson ended on retries: a caller that does not know whether its request was processed, and an idempotency key that makes repeating it safe. This lesson takes that idea and follows it out of a single service into the way services talk to each other, because a queue is where the retry problem becomes the normal case rather than an edge case.

The way you learned first is the synchronous call: service A sends a request to service B and waits for the answer. It is easy to reason about, because the answer arrives in the same breath as the question, and errors are obvious. Its problem is not that it is slow. Its problem is that it couples the two services in time: B must be up and reachable and fast, at the exact moment A needs it, or A fails too. Put three services in a chain and multiply those moments together. If each one is available 99.9 percent of the time, the chain is available 99.7 percent of the time, which is a tenfold increase in downtime for a system whose parts are all individually excellent.

The alternative is to stop waiting. The order service writes the order, publishes a short message saying an order was placed, and returns immediately. Something else, separately, picks the message up and sends the email. The two parts no longer need to be alive at the same moment, and the customer gets an answer in tens of milliseconds. The word for the mechanism is a queue or a broker, and the important shift is not the technology but the direction of the dependency: the order service now depends on the queue accepting a message, which is a small, fast, boring operation, rather than on the mail provider succeeding, which is slow and outside your control.

It is worth being precise about what the queue actually buys, because it is sold as a cure for everything. It buys four things. Decoupling: the producer does not know or care who consumes the message, so you can add a second consumer without touching the producer. Buffering: a spike arrives as a deep queue rather than as a crashed consumer, and the queue drains at whatever rate the consumer can manage. Retries: redelivery is the broker job now, so the producer does not implement backoff. And durability: a good broker keeps the message on disk, so a consumer that restarts does not lose the work that was waiting for it.

SYNCHRONOUS CALL              MESSAGE QUEUE

  caller waits                  caller gets 202 now
  both must be up               consumer may be down
  load hits the callee          queue absorbs the spike
  retry is the caller problem   redelivery is the broker job
  one failure, one error        one failure, a growing depthThe same order that must trigger an email and an invoice, arranged two ways.

Now the part that decides how you write the consumer. Every message system must answer one question: what happens if the consumer crashes between receiving a message and finishing its work? There are only three honest answers. At most once means the broker sends the message and forgets it. If the consumer crashes mid-work, the message is gone, and the customer never gets an invoice. At least once means the broker keeps the message until the consumer acknowledges it, and redelivers if no acknowledgement arrives. The work is never lost, and it may be done twice, because the acknowledgement itself can be lost after the work succeeded. Exactly once is the one everybody asks for, and it is not a delivery mode at all.

AT MOST ONCE           send, forget, never retry
  risk: a lost payment

AT LEAST ONCE          send, retry until acked
  risk: a duplicate payment
  this is what brokers give you

EXACTLY ONCE           not a delivery mode
  it is at-least-once plus a dedup keyTwo real delivery guarantees and one that is really a combination of the second with application logic.

The reason at-least-once is unavoidable is a window you cannot close. The consumer receives the message, does the work, and then acknowledges. If it crashes after the work but before the acknowledgement, the broker has no way to know the work happened, because it never received the acknowledgement. Redelivering is the only safe choice, since the alternative is silently dropping an invoice. You can shrink the window by acknowledging after the work rather than before, but you cannot remove it, because the acknowledgement itself travels over the same unreliable path as everything else. This is not a limitation of a particular broker. RabbitMQ, Kafka and SQS all behave this way, and no configuration flag changes it.

So the practical answer is to make the consumer idempotent, which is the same idempotency key from the previous lesson, moved to the consumer side. The consumer records the message id in the same transaction as the work it performs. If the message is redelivered, inserting the id fails on the primary key, the transaction rolls back, and the second attempt does nothing. It is deliberately boring: one table, one unique constraint, one transaction that does both things together. The order matters enormously. If you record the id and then do the work, a crash in between loses the work forever. If you do the work and then record the id, a crash in between duplicates the work. Only doing both in one atomic step is correct, which is why the table has to live in the same database as the effect.

CREATE TABLE processed (
  message_id  text PRIMARY KEY,
  handled_at  timestamptz NOT NULL DEFAULT now()
);

-- one transaction: record the id and do the work together
BEGIN;
INSERT INTO processed (message_id) VALUES ('9f2c-1');
UPDATE invoices SET sent = true WHERE id = 9001;
COMMIT;

-- a redelivery hits the primary key and rolls backThe structure that makes a redelivery a no-op. Both statements commit together or neither does.

Even an idempotent consumer can fail, and it is important to separate two kinds of failure. A transient failure is a mail provider that is down: retry, and it will probably work. A permanent failure is a message with a malformed body, or referring to an order that was deleted: retry forever and it will never work. Retrying a permanent failure forever is the worst outcome available, because that one message blocks the queue behind it and stops every healthy message too. The standard answer is a bounded retry count with exponential backoff, and after the last attempt the message goes to a dead letter queue: a separate queue that nobody consumes from automatically.

MAIN QUEUE             orders.email
RETRY                  backoff 1 s, 10 s, 100 s, 1000 s
MAX DELIVERIES         5
DEAD LETTER QUEUE      orders.email.dlq
ALERT ON               dlq depth rises at allThe configuration that keeps one poison message from taking the whole pipeline down with it.

Backpressure is the name for what happens when arrivals exceed the rate the consumer can handle, and the queue makes it both easier and more dangerous. Easier, because the producer is not blocked: it keeps accepting orders and the queue grows. More dangerous, for the same reason: nothing pushes back, so the system accepts work it cannot finish, and the only visible symptom is a number that keeps rising. There are two honest responses. Scale the consumers, which works when the spike is over quickly or the work is easy to parallelise. Or refuse new work, which means the producer checks the queue depth and returns a busy response above a threshold. Refusing is unpleasant and is usually the right answer past a certain point, because accepting an order you will deliver next Tuesday is worse service than saying no now.

The number to watch is queue age, not queue depth. Depth alone tells you very little: 50 000 messages is fine if the consumer is draining them in ten seconds, and it is an emergency if the same 50 000 take four hours. Age answers the only question that matters to a customer, which is how long a message published now will wait before somebody handles it. A single number, the age of the oldest unprocessed message, converts a technical backlog into a promise about a delivery time.

One last property, because it changes how you design keys. A queue does not automatically preserve the order in which messages were published, and adding consumers makes it worse: with three workers, two messages about the same order can be handled in either order or at the same time. The usual fix is to route by a key, so that everything about one order always goes to the same partition and is processed in order by one consumer. That key is a design decision with the same flavour as a shard key: choosing the order id gives you per-order ordering, choosing the user id gives you per-user ordering and stronger guarantees across related orders, and choosing nothing at all means you must make every consumer tolerant of messages arriving out of order.