50 minutes. Self-marked against the rubric. 12 marks. In the full course this counts toward the labs and cases share (15% of the course grade). Write your answers in the boxes, then open the model answer and tick each mark you earned honestly. Nothing is sent or stored.
Scenario (invented)
Use the ten-customer table from Lesson 4 (invented):
| Customer | App uses per week (feature) | Renewed? (label) |
|---|---|---|
| 1 | 5 | yes |
| 2 | 4 | yes |
| 3 | 1 | no |
| 4 | 2 | no |
| 5 | 6 | yes |
| 6 | 3 | yes |
| 7 | 0 | no |
| 8 | 4 | no |
| 9 | 2 | yes |
| 10 | 5 | yes |
Rows 1 to 7 are the training set and rows 8 to 10 are the test set. Rule: predict “yes” if the customer uses the app 3 or more times per week.
Tasks
Task 1 (3 marks)
Using the training rows only, state the baseline prediction and its accuracy on the test rows.
Private: nothing you type here is sent or stored.
Task 2 (3 marks)
Score the rule on the training rows and give its accuracy.
Private: nothing you type here is sent or stored.
Task 3 (3 marks)
Score the rule on the test rows and give its accuracy.
Private: nothing you type here is sent or stored.
Task 4 (3 marks)
Write a careful judgement in three sentences: compare the rule with the baseline, comment on the gap between training and test, and state a limit.
Private: nothing you type here is sent or stored.
Rubric and model answer
Task 1: compute the baseline (3 marks)
Model: training rows have four “yes” (1, 2, 5, 6) and three “no” (3, 4, 7), so the baseline predicts “yes”. On the test rows it is wrong for customer 8 and right for 9 and 10: 2 of 3, about 67%.
Task 2: score the rule on training (3 marks)
Model: customers 1, 2, 5, 6 use the app 5, 4, 6, 3 times: predicted yes, correct. Customers 3, 4, 7 use it 1, 2, 0 times: predicted no, correct. All 7 are right, so accuracy is 100%.
Task 3: score the rule on test (3 marks)
Model: customer 8 used the app 4 times, predicted yes, actual no: wrong. Customer 9 used it twice, predicted no, actual yes: wrong. Customer 10 used it 5 times, predicted yes, actual yes: right. Accuracy = 1 ÷ 3 = 33%.
Task 4: judge the result (3 marks)
Model: “On the test rows the rule (33%) did worse than the baseline (67%), although it was perfect on the training rows, a sign of over-fitting. But with only three test rows, the comparison is very noisy. Many more rows are needed before choosing between them.”
Marks so far: 0 of 12