Phase 1: Prepare
Step 1 Warm-up recall
From M1.L3: write the formula for the saving.
Private: nothing you type here is sent or stored.
Step 2 Learning goal
By the end of this lesson you can:
- state the five questions (Foundation);
- explain why accuracy can mislead when events are rare (Core);
- turn a vendor claim into questions (Core).
Phase 2: Teach
Step 3 A familiar example
A vendor says their fraud detector is 99% accurate. It sounds excellent. Whether it is depends on a number they have not given you.
Step 4 Clear explanation
Use these five questions as a standing checklist for any proposal:
| Question | What a good answer sounds like |
|---|---|
| Problem | A named process with a measured cost or delay today. |
| Data | The data exist, are accurate enough and we may legally use them. |
| Evidence | Tested on our own data against a simple baseline, not only on the vendor’s demonstration. |
| Risk | We know the cost of each kind of error and who is affected. |
| Owner | One named executive is accountable for the result. |
The evidence question needs care. Accuracy is the share of cases a system gets right, and it can mislead when one outcome is rare. Suppose 1% of 10,000 transactions are fraud, so 100 cases. A model that always says “not fraud” is right on 9,900 of them, an accuracy of 99%, yet it catches none of the fraud.
So ask for more than accuracy: how many real cases does it catch, how many false alarms does it raise, and what does each kind of mistake cost? Ask also what a simple baseline achieves. If the vendor cannot answer, the number is not decision-grade. The entry on security risk assessment shows a parallel way to weigh such risks.
Step 5 Worked example (4 marks)
A vendor claims 95% accuracy for spotting faulty parts. In your plant 5% of parts are faulty. Respond.
- A model that calls every part “good” would be right on 95%. (1 mark)
- So 95% alone shows nothing beyond that baseline. (1 mark)
- Ask how many faulty parts it catches and how many good parts it rejects. (1 mark)
- Ask to test it on your own parts before any commitment. (1 mark)
Phase 3: Practise
Step 6 Learner practice (instant-marked)
1. The five questions cover problem, data, evidence, risk and:
A named executive must be accountable.
2. 1% of 10,000 transactions are fraud. A model that always says “not fraud” has accuracy:
9,900 of 10,000 are right.
3. How much of that fraud does the always-no model catch?
It never flags fraud.
4. Which is better evidence than a vendor demonstration?
Your data and a baseline give a fair comparison.
Phase 4: Close
Step 7 Recap
- Ask about problem, data, evidence, risk and owner.
- Accuracy misleads when one outcome is rare.
- Ask what it catches, what it wrongly flags and what each error costs.
Step 8 Exit ticket (ungraded)
Rewrite “99% accurate” as three questions you would put to a vendor.
Private: nothing you type here is sent or stored.
Step 9 Go further (Extension)
Read the encyclopedia entry on security risk assessment and note one way to rank risks.
Common mistakes
| Mistake | Why it fails |
|---|---|
| Accepting a headline accuracy figure | Ask for a baseline and the types of error. |
| Skipping the owner question | No owner means no accountability. |
| Relying on the vendor’s data | Test on your own. |
Limits. The fraud figures are invented. Later modules introduce precision and recall, which give a fuller picture than accuracy.