Memorising is not learning

A model can score perfectly on the data it trained on and still be useless tomorrow. Here is how to spot that, and what actually helps.

15 min · 48 XP

Evaluation gave you the tool: a sealed test set and numbers that describe mistakes honestly. This lesson is about the failure that tool exists to catch. A model can be almost perfect on its training rows, awful on everything else, and completely confident about both. The name for it is overfitting.

study hours -> exam score, one row per pupil

100 |                                    *
 90 |                              *
 80 |                        *
 70 |                  *
 60 |            *
 50 |      *
 40 |  *
    +----------------------------------------
      1   2   3   4   5   6   7   8   9

a curve through all six dots
same six dots, no room left overSame six dots, and a curve flexible enough to pass through every single one of them.

The straight line from the earlier lesson missed some dots and got the trend right. Now allow a much more flexible shape and it will thread through all six dots exactly. Every training error becomes zero. Real exam scores were never that tidy: some pupils slept badly, one guessed lucky, one was ill. The curve has not learned the trend, it has learned those accidents. It has memorised the answer key instead of the subject.

+----------------+-----------+-----------------+
| model          | training  | held-back rows  |
+----------------+-----------+-----------------+
| straight line  | 72        | 69              |
| curve, 12 knobs| 100       | 48              |
+----------------+-----------+-----------------+Two models, two kinds of score. The gap between the columns is the story.

That gap between the training score and the held-back score is the whole diagnosis. When a model scores 100 in training and 48 on rows it never met, the flexibility bought nothing and cost a great deal. Notice that the flexible model is not broken and not badly written: it did exactly what it was asked to do, which was to make training errors small. That is why the training score alone can never be the goal.

The cure that works most often is more data. With two examples, a flexible curve can fit anything; with two hundred, each dot pulls only weakly and the curve is forced towards the real trend. A second cure is a simpler model: fewer knobs, less room to memorise accidents. A third is to give the model better features, so that what looked like noise becomes explainable. All three are more useful than any amount of extra training on the same rows.

ice cream sales and the day of the year

nine points, drawn honestly:

sales
   |   *     *  *        *   *
   | *    *         *  *
   +------------------------------
     Apr    May    Jun    Jul

one question the features never asked:

was it raining that afternoon?Nine honest points and one thing the table never recorded.

The two failure modes sit at opposite ends of one dial. Too simple, and the model misses real structure and is wrong everywhere. Too flexible, and it fits the noise and is wrong wherever the noise changes. The useful setting is in between, and the only way to find it is the held-back rows you did not train on. That is the bias and variance tradeoff, and you have just met both halves of it.

complexity -> error

error
  |\                      /
  | \                    /
  |  \                  /  total
  |   \                /
  |    \______________/
  |     \            /
  |      \          /  memorising the noise
  |       \        /
  +------------------------------> knobs in the model
     too simple    about right    too flexible

held-back error at the bottom of the valleyTotal error has a bottom, and it is not at either end of the dial.

Practice

Two models are reported for the same task. Model one: 72 percent on the training rows, 69 percent on rows held back from training. Model two: 100 percent on training rows, 48 percent on held-back rows. Which do you ship, and why?
Hints
- Compare the two columns of the report rather than the highest number in it.
- The second model is also the one with many more knobs to turn.
16 XP
A model predicting whether a student will pass is doing badly in real use. Match every cause with the symptom it produces.
Hints
- Ask who will meet these rows afterwards: the model itself, or the world.
- The second list describes the real world and never appears in any training table.
18 XP
Put the steps of dealing with an overfitted model in order.
1. 1Spot the gap: near perfect in training, much worse on rows it never met
2. 2Make the model simpler, for example fewer features or a less flexible shape
3. 3Keep the simpler model if the held-back score stops improving
4. 4Look at the training score and the held-back score together
5. 5Add more training rows, especially ones that look like the cases that fail
Hints
- You cannot tell a lucky fit from a real one until both numbers stand side by side.
- Making the model smaller is a reaction to a diagnosis, so it comes after the measurement.
18 XP