Train, test and the problem of over-fitting

Step 1 Warm-up recall

From M1.L3: say why a model must beat a baseline.

Your answer

Private: nothing you type here is sent or stored.

Step 2 Learning goal

By the end of this lesson you can:

• split data into training and test sets (Foundation);
• compute accuracy on each (Core);
• explain why training accuracy can mislead (Core).

Step 3 A familiar example

A student memorises last year’s exam paper and scores perfectly on it. Whether she has learned the subject is shown only by a new paper.

Step 4 Clear explanation

If you judge a model on the same examples it learned from, you mostly measure memory. So we divide the data. The training set is used to fit the model. The test set is kept aside and used once, at the end, to score it on cases it has not seen.

Here is an invented table of ten customers: a feature (app uses per week) and a label (renewed or not).

Take a rule: predict “yes” if the customer uses the app 3 or more times per week. Use customers 1 to 7 for training and 8 to 10 for testing. On the training rows the rule is right for all seven (accuracy 100%). On the test rows it is right only for customer 10 (customer 8 used the app 4 times but did not renew; customer 9 used it twice but did renew), so test accuracy is 1 of 3, about 33%.

The baseline from the training rows is “yes” (four renewed, three did not). On the test rows it is right for customers 9 and 10: 2 of 3, about 67%.

The lesson: a model that looks perfect on training data can be worse than the baseline on new data. This is over-fitting. Here the test set is only three rows, so even the test result is noisy; with so little data, no firm conclusion is possible. Real evaluations use far more data. See the entry on Occam’s Razor for why simpler explanations are preferred when evidence is limited.

Step 5 Worked example (4 marks)

A model gets 18 of 20 training rows right and 6 of 10 test rows right.

1. Training accuracy = 18 ÷ 20 = 90%. (1 mark)
2. Test accuracy = 6 ÷ 10 = 60%. (1 mark)
3. The 30-point gap suggests over-fitting. (1 mark)
4. The test figure is the honest estimate of performance on new cases. (1 mark)

Step 6 Learner practice (instant-marked)

Step 7 Recap

• Fit on training data; score once on unseen test data.
• Training accuracy can mislead; over-fitting shows as a gap.
• A tiny test set gives a noisy answer.

Step 8 Exit ticket (ungraded)

A model scores 95% on training data and 62% on test data. Write what you conclude.

Your answer

Private: nothing you type here is sent or stored.

Step 9 Go further (Extension)

Choose a different rule for the customer table and score it on both parts.