Many scientific studies fail to replicate because the statistical threshold used to judge them — usually a p-value below 0.05 — was never designed to certify that a finding is true. It only measures how compatible the data are with a no-effect model. Combine that with journals preferring surprising positive results, and a steady supply of findings that don’t hold up is the predictable outcome.

The scale of it became clear in 2015, and the numbers were worse than most researchers expected.

What the replication studies found

The Open Science Collaboration repeated 100 experimental and correlational studies published in three psychology journals, using high-powered designs and the original materials where they could get them. The results were published in Science in 2015.

Ninety-seven percent of the original studies had reported statistically significant results. Only 36 percent of the replications did. Replication effects were, on average, half the magnitude of the original effects. Other measures told the same story from different angles: 47 percent of original effect sizes fell within the 95 percent confidence interval of the replication, and only 39 percent of effects were judged, subjectively, to have replicated.

One finding points at the cause: replication success was better predicted by the strength of the original evidence than by anything about the teams running the repeats. This wasn’t sloppiness by the replicators.

The threshold is doing work it can’t do

A p-value is the probability, under a specified statistical model, that a summary of the data would be equal to or more extreme than the value actually observed. That’s all it is. It does not measure the probability that the studied hypothesis is true, the probability that the data arose by chance alone, or the size or importance of an effect.

The American Statistical Association issued a formal statement in 2016 spelling this out, because the misreadings had become consequential — and the ASA was direct about the threshold itself: scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific value. The 0.05 line is a convention, not a property of nature. It marks a level of surprise, not a boundary between real and imaginary.

Why the errors pile up in one direction

If the threshold merely added noise, failures would scatter randomly. They don’t — they cluster on the side of overstated effects, and the reason is what happens to results before publication.

Journals have long preferred novel positive findings. A study that clears 0.05 gets published; a study that doesn’t often ends up in a drawer. The published record therefore over-represents results that happened to land on the lucky side of the line. Small samples make this worse: an underpowered study that does reach significance has usually done so by overestimating the effect, which is exactly the pattern the replication project found when effects came back at roughly half size. Analytical flexibility compounds it further — deciding which outliers to exclude, which covariates to include, and when to stop collecting data after seeing results all give researchers many roads to a publishable p-value.

What a failed replication does and doesn’t mean

A failed replication is not proof the original finding was false, any more than the original was proof it was true — both are single estimates with uncertainty attached. Combining original and replication results, and assuming no bias in the originals, left 68 percent with statistically significant effects. Plenty of the underlying science survived; what didn’t survive was the confidence attached to any individual published result.

What changed afterward

The response has been structural rather than statistical. Preregistration commits researchers to a hypothesis and an analysis plan before data collection, removing most analytical flexibility. Registered reports go further, having journals accept a study on the strength of its design before results exist. Larger samples and routine data sharing have become expectations in several fields.

The deeper shift is in how a single result gets read. The ASA’s guidance points toward reporting effect sizes and uncertainty rather than a verdict, and toward treating one study as one piece of evidence among many. That’s a slower way to do science — and it’s the direction the field has moved.