How good is the evidence? Quality, duration and certainty

Step 1 Warm-up recall

From M4.L1: what is publication bias, and what are small-study effects?

Your answer

Private: nothing you type here is sent or stored.

Step 2 Learning goal

By the end of this lesson you can:

• explain what a quality rating of a review tells you (Foundation);
• interpret a difference in effect between shorter and longer interventions (Core);
• grade a claim and state what would raise or lower your confidence (Core).

Step 3 A familiar example

A restaurant has a hundred reviews, but most are written by the owner’s friends. The number sounds impressive and the quality is low. Reviews of studies can have the same problem.

Step 4 Clear explanation

Singh and colleagues (2023) rated the quality of the 97 reviews with the AMSTAR-2 checklist and reported that 77 were rated “critically low”. They also noted that most of the evidence concerned mild-to-moderate depression, and that the lack of participant blinding in exercise trials may produce expectancy effects. They graded the certainty of the evidence for depression and anxiety as high (grade A) and for psychological distress as lower (grade B), with the distress finding resting on a small number of reviews.

Duration. The authors reported that shorter interventions (up to 12 weeks) had a larger median effect on depression (−0.84) than longer ones (24 weeks or more, −0.28), and that all durations were effective. Why might a shorter programme look better? Possible reasons include novelty and enthusiasm fading, different people volunteering for short and long studies, and chance. The data cannot tell you which; an overview of reviews does not randomise duration.

Putting this together: the headline numbers are encouraging, but the quality ratings, the usual-care comparison and the missing blinding all lower confidence in the exact size. Your evidence card should say so.

Step 5 Worked example (4 marks)

Grade the claim “exercise reduces depressive symptoms”.

1. Strong point: 97 reviews, 1,039 trials, consistent direction, no publication bias signal in funnel plots. (1 mark)
2. Weak point: 77 of 97 reviews rated critically low in quality. (1 mark)
3. Other limits: usual-care comparison, no blinding, mostly mild-to-moderate depression. (1 mark)
4. Grade: Moderate, with limits stated, because the direction is consistent but the quality and size are uncertain. (1 mark)

Step 6 Learner practice (instant-marked)

Step 7 Recap

• 77 of 97 reviews were rated critically low in quality.
• Shorter interventions had a larger median effect on depression (−0.84) than longer ones (−0.28); the reasons are not settled.
• Grade a claim on consistency, quality, size and limits.

Step 8 Exit ticket (ungraded)

Give your grade for “exercise improves mood” and the one fact that would most change it.

Your answer

Private: nothing you type here is sent or stored.

Step 9 Go further (Extension)

Look up the AMSTAR-2 checklist and find two of its critical items.