A straight line through the points

Last lesson you saw what a model is. This lesson opens one up. We use the simplest model that exists, a straight line, because you can follow every number inside it by hand.

study hours -> exam score

100 |                                    *
 90 |                              *
 80 |                        *
 70 |                  *
 60 |            *
 50 |      *
 40 |  *
    +----------------------------------------
      1   2   3   4   5   6   7   8   9

six pupils, one dot eachSix pupils, one dot each: hours studied across, exam score up.

Look at those dots. More hours of study go with a higher score, roughly, and roughly is the important word. The dots do not sit on any perfect line. A model of this is a line that passes as close to all six dots as one straight line can. When you have several features instead of one, the same sentence becomes a flat surface, or a surface in more dimensions than you can picture. Nothing else changes: features in, one number out.

The line has two numbers to choose. The weight says how much one extra unit of a feature moves the prediction. The other number is where the line starts when the feature is zero, and you can think of it as the background level. Together they are the parameters of the model, the numbers training is allowed to touch. A weight is influence, and that is genuinely useful information: it tells you which feature pushes the answer hardest, and in which direction.

score = start + weight * hours

try 1:  score = 30 + 6 * hours
   4 h -> 54    real 71    miss 17
   8 h -> 78    real 86    miss 8

try 2:  score = 15 + 10 * hours
   4 h -> 55    real 71    miss 16
   8 h -> 95    real 86    miss 9

try 3:  score = 40 + 5 * hours
   4 h -> 60    real 71    miss 11
   8 h -> 80    real 86    miss 6Three candidate lines, and how far each guess lands from the real exam scores.

How do we compare two lines? For every training row we take the real label, subtract the prediction, and that difference is the error on that row. One row being eleven points off is exactly as bad as another row being eleven points off the other way, so we square each error before adding them up. Squaring also punishes one huge miss more than several small ones. The total is the cost, and training means searching for the two numbers that make it as small as possible.

score = 40 + 5 * hours

+-------------+-------+--------+-------+--------+
| study_hours | score | guess  | error | square |
+-------------+-------+--------+-------+--------+
| 3           | 55    | 55     | 0     | 0      |
| 4           | 71    | 60     | 11    | 121    |
| 5           | 70    | 65     | 5     | 25     |
| 7           | 86    | 75     | 11    | 121    |
+-------------+-------+--------+-------+--------+

sum of squares: 0 + 121 + 25 + 121 = 267One line, four rows, and the cost it has to pay for being wrong.

Why not just add the errors as they are? Because a miss of plus ten and a miss of minus ten would cancel out and look like a perfect model. Squaring keeps every miss visible and makes the total a single honest number. Different shapes use slightly different costs, but the idea never changes: choose parameters, score them, lower the score.