Probability: reasoning under uncertainty

Up to here every claim in this course was true or false. Probability is what you use when the honest answer is a number between them, and the discipline it imposes is simple: before you compute anything, write down the sample space. That is the list of everything that can happen, with exactly one entry per outcome. For two coin flips the space below has four entries, not three. The tempting mistake is to write two heads, one head, no heads, which collapses two distinct outcomes into one and makes every later calculation wrong by a factor of two.

two fair coins

  S  =  { HH, HT, TH, TT }

  P(HH)             = 1 / 4
  P(at least one H) = 3 / 4
  P(exactly one H)  = 2 / 4

  P(HH) + P(HT) + P(TH) + P(TT)  =  1

This is where the previous lesson pays off. When every outcome in the sample space is equally likely, a probability is just a count divided by a count, and C of n comma r and P of n comma r do the arithmetic. The whole of the next section is counting favourable outcomes and dividing by the size of the space, which is why combinatorics came first.

/* independent */

  P(A and B)  =  P(A) * P(B)

  1/2 * 1/2   =  1/4

/* dependent, cards, no replacement */

  P(first ace)             =  4 / 52
  P(second ace | first ace) =  3 / 51

Two events are independent when knowing that one happened tells you nothing about the other, and only then do their probabilities multiply. Flipping a coin twice is independent. Drawing two cards without replacement is not, because the first draw removed a card from the deck: the chance of the second ace is three in fifty-one, not four in fifty-two. The classic confusion is thinking that independence is about whether the events are conceptually unrelated. It is not, it is about whether the first changed the numbers for the second. In software the same distinction appears as sampling with and without replacement, and a surprising number of broken simulations come from sampling without replacement while assuming they did not.

P(A|B)  =  P(A and B) / P(B)

prevalence 1 / 1000        sensitivity 0.99        specificity 0.95

population 1000000

  ill                    =      1000
  healthy                =    999000

  ill and flagged        =  1000    * 0.99  =      990
  healthy and flagged    =  999000  * 0.05  =    49950
  flagged                =                     50940

  P(ill | flagged)       =  990 / 50940        =  0.019

Now the part that most people, including most doctors and most engineers, get wrong. Read the block below slowly. One person in a thousand has the disease. The test catches 99 of every 100 ill people, and flags 5 of every 100 healthy ones, which sounds like a very good test. Follow the population of a million through it. Nine hundred and ninety of the thousand ill people are correctly flagged, and almost fifty thousand healthy people are wrongly flagged. Now count: of about fifty-one thousand positive results, only nine hundred and ninety are real. That is under two percent. The test is accurate and the result is still almost certainly a false alarm, and both statements are true at once.

P(positive | ill)  =  0.99
P(ill | positive)  =  0.019

ratio  =  0.99 / 0.019  =  52

  990 / 50940  =  0.019

The reason is not arithmetic, it is that two different questions were silently swapped. The sensitivity, 0.99, is the probability of a positive result given that you are ill. The number that matters to a patient is the opposite: the probability that you are ill given a positive result. They are not the same number, they are not even close, and they differ here by a factor of about fifty-two. This swap has a name, the base rate fallacy, and it is not a curiosity for exams. It drives unnecessary treatment, needless panic and bad product decisions, and it survives in intelligent people because the correct answer feels wrong. The general lesson is worth carrying everywhere: whenever you see a conditional probability in a sentence, ask which event is on which side of the bar, and whether the rare thing is in the numerator or the denominator.

If you do not believe the arithmetic, simulate it. The code below runs a million people through the same numbers and prints the fraction of positives that are genuinely ill. It will land near 0.019 every time.

import random

ill = 0
flagged = 0
true_positive = 0

for _ in range(1000000):
    sick = random.random() < 0.001
    hit = random.random() < (0.99 if sick else 0.05)
    if sick:
        ill += 1
    if hit:
        flagged += 1
        if sick:
            true_positive += 1

print(true_positive / flagged)Monte Carlo check of the medical test numbers: 0.001 prevalence, 0.99 sensitivity, 0.05 false positive rate.

Expected value is the single number that summarises a random outcome, and it is the average you would get per trial over many trials. Multiply each outcome by its probability and add. The lottery ticket below has an expected value of minus one, which is the honest way to say that the ticket costs one more than it returns on average. Expected value is also what makes the test question from earlier concrete: a test is only worth running when the expected benefit of a correct result outweighs the cost of the false alarms it produces, and for a rare disease with an imperfect test that is a real calculation, not an opinion.

/* lottery ticket, price 2 */

E  =  (1/1000000) * 1000000  +  (999999/1000000) * 0  -  2
   =  1 - 2
   =  -1

This connects directly to model evaluation from tier 1, and it is the same failure wearing different clothes. Accuracy is a conditional probability read backwards: it tells you how often the model is right overall, while what you usually need is how often the model is right when it says yes. On a rare class the two numbers are wildly different, which is why precision and recall exist and why a model with 99 percent accuracy can be worthless. The base rate is not a detail of the metric, it is the thing that decides whether the metric means anything at all.