Error and the baseline: what must a model beat?

Step 1 Warm-up recall

From M1.L2: name the label in a table that predicts whether a customer renews.

Your answer

Private: nothing you type here is sent or stored.

Step 2 Learning goal

By the end of this lesson you can:

• calculate absolute error and mean absolute error (Foundation);
• build a baseline that predicts the mean (Core);
• say why a model must beat its baseline (Core).

Step 3 A familiar example

If you forecast tomorrow’s weather as “same as today”, you are right surprisingly often. Any forecasting system must at least do better than that simple habit.

Step 4 Clear explanation

The error of one prediction is the gap between the prediction and the true value. The absolute error ignores direction: |prediction − truth|. The mean absolute error (MAE) is the average of the absolute errors over all examples.

A baseline is a very simple model that any real model must beat. For a number such as price, a common baseline always predicts the mean of the known values. For a category such as yes/no, it always predicts the most common class. Consider five invented prices in thousands of pounds: 200, 220, 240, 260, 280. Their mean is 240. The baseline predicts 240 for every home.

The absolute errors are 40, 20, 0, 20 and 40, which sum to 120. Dividing by 5 gives an MAE of 24. A model with an MAE above 24 would be doing worse than simply quoting the mean. A model with an MAE of 10 would be a real improvement. Without a baseline, a number such as “MAE 24” means nothing.

Step 5 Worked example (4 marks)

Four true values are 10, 12, 14, 20 and a model predicts 11, 11, 16, 18.

1. Absolute errors: 1, 1, 2, 2. (1 mark)
2. Sum = 6. (1 mark)
3. MAE = 6 ÷ 4 = 1.5. (1 mark)
4. The baseline (mean 14) would give errors 4, 2, 0, 6, so MAE = 12 ÷ 4 = 3; the model beats it. (1 mark)

Step 6 Learner practice (instant-marked)

Step 7 Recap

• Absolute error = |prediction − truth|; MAE is their mean.
• A baseline predicts the mean (numbers) or the most common class (categories).
• A model must beat its baseline to be worth using.

Step 8 Exit ticket (ungraded)

Compute the MAE of a baseline that predicts the mean for the values 4, 6, 8, 10, 12.

Your answer

Private: nothing you type here is sent or stored.

Step 9 Go further (Extension)

Think of a simple habit that is a strong baseline for a forecast in your own field.