Scaling: more load, more machines, more problems

In architecture:1 you built a system out of a client, a server, an API contract, a cache and a monolith. It worked. This lesson is about the day it works for ten thousand people instead of ten. Nothing here is a new concept: it is what the concepts from architecture:1 and os:2 cost when you take them seriously.

There are exactly two directions to grow. Vertical means a bigger machine: more cores, more RAM, faster disks. Horizontal means more machines of the same size. Vertical is genuinely underrated. It requires no code change, it keeps every assumption you already made true, and a single modern server with 64 cores and 512 GB of RAM serves a shocking amount of traffic. A blog, a small shop, an internal tool and most B2B products never need anything else. The trouble with vertical is that it is a ladder with a top rung: at some point the biggest machine you can rent is not enough, the price per core stops being linear and starts doubling, and replacing that one machine is a maintenance window rather than a rolling update.

VERTICAL                      HORIZONTAL

  1 x 32 cores                  32 x 1 core

  limit: silicon                limit: coordination
  cost: superlinear             cost: linear plus a tax
  failure: one box               failure: 1 of 32 lost
  restart: minutes of downtime  restart: drain and replace
  ceiling: ~ a few hundred GB   ceiling: none in practiceBoth directions grow the same system. They fail differently, and that difference is usually what decides it.

Here is the part that decides everything, and it is the same idea as stateless HTTP from architecture:1, now seen from the other side. A stateless service keeps nothing between requests: each one arrives, gets an answer, and the process forgets it. Such a service scales by changing a number. Run three copies or thirty, and any copy can answer any request. A stateful service remembers something that belongs to a specific user, and that memory is pinned to one machine. Now adding a machine does not help by itself: you also have to decide where the state lives, how the new machine learns about it, and what happens to it when the machine holding it dies. That is the whole difficulty of scaling in one sentence: computers are easy to multiply, memory of a conversation is not.

The classic example is the shopping cart. If the cart lives in the memory of the server process, then the user must keep hitting that same process, or the cart empties itself. The fix is to move the state out of the process into something shared: a database row, a Redis key, or a signed cookie the client carries. Once the cart lives in shared storage, the service in front of it is stateless again, and it multiplies like any other. Notice the shape of the move: you did not make the state disappear, you moved it to the one place that is allowed to have state. That is what the database is for, and it is also why the database becomes the next bottleneck.

STATELESS SERVICE              STATEFUL SERVICE

  any replica answers            only the owner answers
  add a pod: instant             add a node: rebalance keys
  kill a pod: no loss            kill a node: lose its shard
  scale by count                 scale by partition

  cart lives in a shared store, not in the processThe same service, once its state has been moved out of the process and into a shared store.

Once you have several identical replicas, something has to choose which one gets each request. That something is a load balancer, and the interesting part is not the algorithm but the choice of what to balance on. Round robin sends request one to replica one, request two to replica two, and so on: simple, and fine when every request costs about the same. Least connections sends the next request to whichever replica currently has the fewest in flight: better when requests vary a lot in cost, because one slow request will not keep pulling traffic onto an already busy replica. Hashing takes a value from the request and maps it to a replica, which is the only one of the three that gives you stickiness.

A load balancer is only as useful as its idea of which replicas are healthy. The shallow check is a TCP connect: if the port accepts, the replica is considered up. That is nearly worthless, because a process can accept connections while its database pool is exhausted and every request times out. The useful check is a real endpoint that verifies the dependencies the replica needs to do its job, and returns failure when they are missing. Even then, be careful about ordering: a health check that fails too eagerly takes a healthy replica out during a brief database blip, and if it fails on all of them at once you have turned a slowdown into a full outage.

There is a distinction that reorganizes the whole problem: reads and writes do not scale the same way, and they do not have to scale together. Most systems read far more than they write. A news site might serve a hundred thousand page views for every article published; a shop might show a product ten thousand times for every time its stock changes. Reads are also naturally parallel: ten different users reading the same row have no reason to conflict. Writes are the opposite, because they change shared state and something has to arbitrate the order. So the standard first move is to split them: one primary that accepts writes, and several replicas that copy it and serve reads. Notice that this is a trade, not a free lunch. You have just bought read capacity, and the price is that a read may now return a value that is slightly out of date.

-- writes go to the primary
INSERT INTO orders (id, user_id, total)
VALUES (9001, 42, 12.90);

-- reads may go to a replica, up to 200 ms behind
SELECT id, total
FROM   orders
WHERE  user_id = 42
ORDER  BY id DESC
LIMIT  20;

-- one connection per role, chosen explicitly
-- write pool -> primary.internal:5432
-- read pool  -> replica.internal:5432Splitting reads from writes is a few lines of configuration and a new class of bug.

Replication lag is the delay between a write landing on the primary and becoming visible on a replica. Under light load it is a millisecond or two and nobody notices. Under heavy load it grows, because the primary is busy serving writes and the replica has to apply them one at a time in order. Numbers worth carrying around: a healthy PostgreSQL streaming replica sits under about 10 ms; a busy one under a write spike can reach several hundred milliseconds; a replica that has fallen behind on a large batch or a schema change can be minutes behind and still be reported as up. Lag is not a bug to fix, it is a property of copying data over a network onto a second machine that is also doing work.

The bug lag creates is called read-your-own-writes, and it is the single most common way a read replica produces a support ticket. A user saves a change, the write goes to the primary and succeeds, the page reloads, the read goes to a replica that has not caught up, and the user sees their own old data. From the the user side the save simply did not work, so they do it again. The honest fix is to notice that the user does not need the newest data in general, they need their own writes to be visible to themselves. So route reads to the primary for a short window after that user writes: a few seconds, or until the replica reports that it has caught up past that position of that write. Note what this costs: you give up some read capacity, for the subset of reads that follow a write by the same user. That is usually a small subset, and the trade is worth it.

Replication copies the whole dataset to every replica, so it scales reads. It does nothing for writes, because every replica still has to apply every write. When writes are the bottleneck, the only remaining move is to stop putting all the data in one place: split the data itself into pieces, called shards, each living on a different machine, and each handling the writes for its own piece. Now a write touches one shard, and ten shards can absorb ten times the write traffic. This is the point where a single logical database becomes a fleet you have to route to yourself, and where the application starts containing code that answers the question which shard holds this row.

KEY            DISTRIBUTION     HOT SPOT          CROSS-SHARD QUERY

  user_id        even             none              joins on user_id only
  created_at     even over time   today only        every range scan
  country        skewed           one big country   any global report
  auto id        even at write    none              no locality at allFour candidate shard keys for an orders table, and the price each one charges.

The shard key is the decision that is nearly irreversible, and it is worth understanding why before choosing one. Every query the application makes now has to either name the shard key or be sent to every shard and have its results merged, which is dramatically slower and quickly becomes the dominant cost. So the shard key silently decides which queries will remain fast for the next five years. Choose user id for an orders table and every query by user is a single-shard lookup, while a report over all orders must scatter and gather. Choose created_at and every report by month is fast, but all of the writes of today land on one shard, which is exactly the shard that is already hottest. Changing the key later means reading every row out of every shard and writing it back in a different arrangement, with the system live and serving traffic the whole time.