Observability: finding out what the system is doing

The previous lesson ended with a warning that fits this one exactly: a queue hides failure. If the email worker dies, nothing returns an error, and the customer who notices first is the one you least want to hear it from. Everything in this lesson is the answer to that problem: how a running system tells you what it is doing, without a user having to complain first.

There are three kinds of telemetry, and the useful way to hold them is by the question each one answers. Metrics are numbers sampled over time: requests per second, error rate, queue age, memory used. They are cheap to store and cheap to query, and they are the only one that scales to answering is anything wrong right now. Logs are individual events with detail: this request failed with this error, for this user, at this time. They are expensive, and they are the only one that tells you what exactly was different about the one request that broke. Traces follow one request across every service it touched, recording how long each step took, and they are the only one that answers where the time went.

PILLAR     SHAPE                ANSWERS                        COST

  metric     number over time     is it bad, and since when     cheap
  log        event with fields    what exactly happened here    medium
  trace      spans with parents   where did the 900 ms go       sampledThe three pillars, the question each one answers best, and what it costs to keep.

The reason logs alone do not work is arithmetic, and it is worth doing once so the number sticks. Forty application pods each writing five hundred lines per second is twenty thousand lines per second. Over one hour that is seventy-two million lines, and at two hundred bytes each it is about fourteen gigabytes per hour, or roughly three hundred gigabytes a day. Nobody reads that. You cannot find a needle by reading a haystack whose size doubles while you read it. This is why logs are a detail store you query with a specific question in mind, not a thing you look at.

40 pods x 500 lines/s = 20000 lines/s

  1 hour   =  72 000 000 lines
  at 200 bytes each  =  14 GB per hour

  reading by eye: never
  grep across 40 files: minutesWhy nobody looks at raw logs, and why the answer is to query them rather than browse them.

The fix for logs is to stop writing sentences and start writing records. An unstructured line like order 9001 failed for user 42 after 903 ms is readable by a human and nearly useless to a machine: extracting the order id means writing a regular expression that breaks the moment somebody changes the wording. A structured record writes each value under a name, so the log system can filter by order id, group by error type, and count. This costs almost nothing to adopt and it converts logs from something you read into something you query. The one discipline that matters is agreeing the names, because user_id in one service and uid in another makes the join impossible exactly when you need it.

{
  "ts": "2026-10-13T09:14:22.481Z",
  "level": "error",
  "event": "payment_declined",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "user_id": 42,
  "order_id": 9001,
  "amount_cents": 1290,
  "duration_ms": 903
}One line of structured log, with the fields that make it queryable and the trace id that connects it to everything else.

The distinction that reorganizes monitoring is between a symptom and a cause. A symptom is something a user experiences: checkout is slow, login fails, the page is blank. A cause is a mechanism inside your system: the CPU is at ninety percent, a lock is held, a connection pool is exhausted. Causes are what you can fix, which is why people alert on them. That instinct is wrong, and the reason is that most causes are harmless most of the time. CPU at ninety percent on a busy day is a system working well. A lock wait of six seconds is catastrophic. The number is the same kind of thing; only the user-visible effect tells you which one you are looking at. Alert on the symptom, and use the cause as the explanation you go looking for once the symptom fires.

So the rule is: page a human when a user is being harmed, and let everything else be a dashboard or a ticket. Concretely, that means the checkout error rate crossing two percent is worth waking somebody, and the CPU being at eighty percent is not, because the second one might be entirely fine and it certainly does not by itself mean anybody is suffering. This is not a matter of taste. A page is a claim that something is wrong right now and needs a person. If you page on causes, you will page on things that are not wrong, engineers will start ignoring the pager, and the one alert that mattered will sit unread while the site is down.

ALERT ON (symptom)            DO NOT PAGE ON (cause)

  checkout error rate > 2%      cpu above 80 percent
  p99 checkout latency > 2 s    disk above 75 percent
  queue age > 5 minutes         one pod restarted
  login success rate < 99%      memory above 70 percent

  a cause that hurts nobody pages nobodyThe left column is worth waking someone for. The right column is worth a graph on a screen.

The last idea is the one that turns all of this into a number you can argue with. An SLI is an indicator: a proportion you can measure, expressed as good events divided by valid events, measured at the point where the user experiences it. For a checkout, that means successful checkouts divided by all attempted checkouts, counted at the edge of the system rather than inside some healthy internal service. The measurement point matters enormously. A service can report a hundred percent success while the load balancer in front of it is timing out half the requests, and only the edge measurement sees that.

SLI   good requests / valid requests
      measured at the edge, not inside

SLO   99.9 percent over 30 days

BUDGET
  30 days          =  43200 minutes
  allowed bad      =  43.2 minutes
  burn 10x         =  page now
  burn 1x          =  ticket tomorrowThe arithmetic of a 99.9 percent objective over thirty days, and the two speeds of burn worth reacting to.

An SLO is a target for that indicator, like 99.9 percent over thirty days, and the error budget is what remains: the amount of failure you have chosen to tolerate. Thirty days is 43 200 minutes, so 99.9 percent leaves 43.2 minutes of allowed badness, which is a number you can actually spend. The budget changes conversations. When it is healthy, the team can deploy quickly and take risks, because the point of the budget is to be used. When it is nearly gone, deploys get more careful, and when it is exhausted the standard move is to freeze feature work and spend the time on reliability. That is the whole mechanism, and it replaces an argument about feelings with a decision about a number.

This is the answer to the oldest sentence in software. It works on my machine is not a statement about the code, it is a statement about two different environments. The production request goes through a different network path, a different dataset, a different configuration and a different concurrent load. And it is a measurement problem because the failing request was never captured with enough context to compare: which user, which version of the code, which replica, which database row. The observability answer is not to argue about the sentence, it is to make it unnecessary. If every request carries a trace id and every log line carries the same id, then one example of the failing request from production is worth more than an hour of reproduction attempts, because it contains the facts instead of a hypothesis.

One practical consequence is worth stating because it changes how you behave during an incident. The first useful question is never what should we change, it is what do we know. A trace id from a report, a template of the failing request, the exact moment it started, and whether it affects everyone or a subset: those four facts usually narrow the search enough that the fix is obvious. Teams that skip this step spend the incident changing things, which destroys the evidence for the next attempt at the same problem, and then the incident repeats a month later with nobody able to say whether it is the same one.