Consistency: what you give up when data lives in two places

The previous lesson ended on an uncomfortable note. You added read replicas to handle more traffic, and the very first thing you bought was a second copy of the truth that can disagree with the first one. This lesson is about that disagreement: when it happens, what it costs, and how to keep it from reaching a user as a bug.

The moment data exists on two machines, a question appears that did not exist before: which copy is right? On one machine the answer is trivial, because there is only one. On two machines you have to decide two separate things. First, what a read is allowed to return while a write is still spreading. Second, what to do when the machines cannot talk to each other at all. Both decisions have real costs, and neither can be avoided by choosing better hardware. This is the subject people argue about under the name CAP, so it is worth getting the argument exactly right.

You have probably heard CAP as pick two of three: Consistency, Availability, Partition tolerance. That version is memorable, is repeated everywhere, and is misleading in a way that leads to bad decisions. Read literally, it suggests you can choose consistency and availability and simply skip partition tolerance. You cannot. A partition is not a setting you decline; it is a network cable that gets unplugged, a switch that reboots, a datacentre link that saturates, a garbage collector pause long enough that the other machine gives up. If your system runs on more than one machine connected by a network, partitions will happen, and partition tolerance is not on the menu.

Notice what each branch actually costs, because they fail in completely different ways. Refusing to answer is loud: a user sees an error page, the checkout button returns 503, and you find out about it from your monitoring within seconds. Letting copies diverge is quiet: every request succeeds, the site is fast, and two users are looking at two different balances with no indication that anything is wrong. Which one you want depends on the feature, not on the company. For a bank transfer the loud failure is obviously correct, because an answer that is confidently wrong about the money of somebody is worse than no answer. For a like counter or a view count the quiet failure is obviously correct, because nobody is harmed by a count being one off for two seconds.

WHEN THE LINK BREAKS, PICK ONE BEHAVIOUR

  refuse the write        A returns 503 until the link returns
  accept and diverge      A returns 200, B stays stale
  accept a quorum only    A needs 2 of 3 nodes to answer
  last write wins         conflicts resolved by timestamp

  the partition is never optionalThe four real options during a partition. Notice that three of them are variations of refusing or diverging, not a third axis.

One more correction matters, because it changes design decisions every day. Consistency is not only about failures; it also costs latency even when everything is healthy. Strong consistency means a write is not finished until enough copies agree it happened. If those copies are in three datacentres on three continents, and the network round trip between them is 70 ms, then a write that must be acknowledged by a majority pays for that distance on every single request. This is why the CAP tradeoff is sometimes restated as a choice between consistency and latency even when nothing is broken: agreeing takes time proportional to the distance between the copies, and distance is not negotiable.

Now the two everyday words. Strong consistency means that once a write is acknowledged, every later read anywhere sees it. There is no moment in which a correct read can return the old value. The price is that every read either waits for agreement or is routed through the one place that has authority, which is exactly as slow and as fragile as it sounds. Eventual consistency means the opposite promise, and it is worth stating precisely because it is usually stated badly: after a write stops, the copies converge, and if no new writes arrive they will agree within some bounded time. There is no promise about what a read sees in the meantime. That gap is not a bug in the model, it is the model.

BANK BALANCE                     LIKE COUNTER

  money leaves an account          a stranger taps a heart
  a wrong read costs money         a wrong read costs nothing
  must be strong                   eventual is fine
  cost: one writer at a time       cost: two users see 41 vs 42
  reject a write rather than lie   show 41 now, 42 in 800 msThe same distributed system, holding both kinds of data at once, with two different promises about them.

The important consequence is that consistency is not a property of the system, it is a property of the data. The same application can hold a bank balance under strong consistency and a like counter under eventual consistency, and nobody finds that strange once it is said out loud. Ask what a wrong answer costs. If a stale read of an account balance could let somebody withdraw money twice, that read must be strong, and it is worth paying the latency and the availability to get it. If a stale read of a like count shows 41 instead of 42 for eight hundred milliseconds, the cost is that nobody notices, and paying for strong consistency there would be pure waste. Choosing per feature is the whole skill, and it is a business judgement expressed as a technical one.

There is a middle option between strong and eventual that solves the majority of real complaints, and you met it in the previous lesson: read-your-own-writes. It is weaker than strong consistency, because other users may see the old value, and stronger than plain eventual, because the person who made the change always sees it. It costs one routing rule: reads from a user who wrote recently go to the primary. Most products need exactly this and nothing more, which is worth remembering when somebody proposes making the whole system strongly consistent to fix a profile page.

Now the part that ties everything together, and it is the concept that separates a working distributed system from a demo. Every network call can fail in a way where you do not know what happened. The request may have been lost on the way out, or it may have been processed and the response lost on the way back. From the client those two cases look identical: a timeout. So the client retries, and if the first attempt actually succeeded, the retry is a second real payment, a second email, a second shipment. Retrying is not optional either, because not retrying turns every transient blip into a user-visible failure. This is the same reasoning as at-least-once delivery, and it applies to every RPC, not only to message queues.

CREATE TABLE payments (
  id           bigserial PRIMARY KEY,
  idem_key     text NOT NULL UNIQUE,
  account_id   bigint NOT NULL,
  amount_cents integer NOT NULL
);

-- the retry sends the same idem_key, so the second insert loses
INSERT INTO payments (idem_key, account_id, amount_cents)
VALUES ('pay-9f2c-1', 42, 1290)
ON CONFLICT (idem_key) DO NOTHING;The only structure that makes a retried write safe: the client supplies a key, and the database enforces that the key is used once.

There is a shortcut that avoids some of the machinery: make the operation idempotent by construction. Setting a value to a given state, like status equals shipped, can be repeated any number of times with the same result, so it needs no key. Adding to a value, like balance equals balance plus 100, cannot, because repeating it changes the result. Notice that this is a property of the operation, not of the code that calls it. Where the business rule allows it, prefer the operation that can be repeated safely, and mark the ones that cannot with an explicit idempotency key, so that the unsafe case is a deliberate, visible choice rather than an oversight.

The last hard problem is making two separate systems change together. Inside one database, a transaction gives you atomicity: several statements succeed or all of them are rolled back, and you never see a half-applied state. Across two services with two databases, that guarantee is not available in the same form. The classic protocol, two-phase commit, has a coordinator ask every participant to prepare, and then tell them all to commit. It works, and it has a famous weakness: the coordinator is a single point of failure, and if it dies after the prepare phase, participants are left holding locks and waiting, sometimes for a long time. That is why its use is mostly limited to systems that can afford the delay and the coordination cost.

The practical alternative is to accept that there will be states in between, and to design for them. A saga breaks the work into local transactions, one per service, each of which commits on its own, and a sequence of compensating actions that undo the earlier steps when a later one fails. Placing an order becomes: reserve the stock, charge the card, and if creating the shipment fails, refund the card and release the stock. Notice what this gives up honestly. There is a window in which the money is taken and the order is not yet confirmed, and users can see that window unless you hide it behind a status like processing. That is a real cost, and the benefit is that no service holds a lock while waiting for another, so no single failure freezes the whole flow. Most systems that need this reach for the saga, because it fails in small, recoverable pieces instead of stalling in one large one.