The Replication Crisis: Why So Much Published Science Doesn’t Hold Up on a Second Look

In 2015, a team of 270 researchers set out to do something almost nobody in science actually does: repeat someone else’s experiment, exactly, and see what happens. They picked 100 psychology studies already published in three top journals, followed the original methods as closely as the original authors would allow, and ran every one of them again. If published science worked the way most people assume it does, nearly all of those studies should have produced the same result a second time. Instead, barely more than a third did. That project did not invent the worry that published findings sometimes don’t hold up — researchers had quietly flagged it for years — but it gave the problem a number, a methodology, and a name that has stuck ever since: the replication crisis.

What Is the Replication Crisis, Exactly?

The replication crisis refers to the documented pattern, found across multiple fields but most rigorously measured in psychology, in which a large share of published, peer-reviewed findings fail to produce the same result when independent researchers repeat the original study under closely matched conditions. It is not primarily a story about fraud, though individual fraud cases exist and have drawn headlines of their own. It is a story about something more unsettling: large numbers of technically legitimate, properly reviewed studies whose reported effects turn out to be smaller, shakier, or simply absent the second time anyone looks.

Specialty Digest’s companion explainer on why studies fail to replicate covers the basic mechanics; this piece goes further into how big the problem actually is, what’s driving it, and what science is doing about it.

What Did the Landmark 2015 Study Actually Find?

The project described above is the Open Science Collaboration’s “Reproducibility Project: Psychology,” published in the journal Science in 2015. Of the 100 original studies selected from Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, 97 had reported a statistically significant result. When the replication team reran them — using original materials where available, with protocols reviewed by the original authors, and with enough statistical power to detect the original effect roughly 92% of the time — only 35 of the 97 replications, or 36%, reached statistical significance in the same direction.

The effect sizes told an even starker story. The average strength of the original findings was roughly double the average strength of the replicated findings, and in 83% of the paired studies, the original effect was stronger than the replication. Even when the team pooled the original and replication results together in a combined analysis, only 68% of the studied effects could be said with confidence to differ from zero at all. One pattern stood out as a clue: replication success tracked much more closely with how statistically strong the original evidence had been than with how experienced or careful the replicating team was — suggesting the problem lives upstream, in what gets published in the first place, not in sloppy copying downstream.

Is This Only a Psychology Problem?

No, though psychology is where the issue has been most systematically measured. A 2016 survey by the journal Nature, which drew more than 1,500 responses from researchers across disciplines, found that 70% had at some point failed to reproduce another scientist’s results, and 50% had failed to reproduce one of their own findings. More than half of respondents said they believed there was a significant reproducibility crisis in science generally, and another 38% called it a slight crisis — leaving a clear majority who saw a real problem, even if they disagreed on its size. Asked why, the two most commonly cited reasons were pressure to publish and selective reporting of results, followed by insufficient replication within the original lab, inadequate oversight or mentorship, and low statistical power.

Why Do So Many Findings Fail to Hold Up?

Several mechanisms compound each other. Journals have historically favored novel, statistically significant, surprising results over null or modest ones, which creates publication bias: the findings that make it into print are a selected, inflated sample of everything that was actually tested. Researchers facing pressure to publish can also, often without any intent to deceive, make a series of small, defensible-seeming analytical choices — which variables to include, which outliers to drop, when to stop collecting data — that nudge a borderline result over the line into statistical significance, a pattern researchers call “p-hacking” or, more neutrally, “researcher degrees of freedom.” Add studies that are simply too small to reliably detect the effect they claim to find, and the result is a literature in which a meaningful share of published effects are, statistically speaking, closer to noise than researchers realized at the time.

What Is Being Done to Fix It?

The response has reshaped how parts of science are done. Preregistration — publicly committing to a study’s hypothesis, methods, and analysis plan before collecting data — closes off the ability to quietly adjust the analysis after seeing the results. A related format called the registered report, promoted by organizations including the nonprofit Center for Open Science, has journals review and conditionally accept a study’s design before the outcome is known, which removes the incentive to bury an unexciting result. Large, multi-lab collaborative replication projects — following the model the 2015 psychology project itself established — have become a recognized way to stress-test influential findings before the field builds further theory on top of them. None of this has eliminated the problem, but it has made the scale of it visible and measurable in a way that simply wasn’t true fifteen years ago.

Evidence Strength

Well-supported: That a substantial share of published findings — particularly in psychology, and by researchers’ own self-report across other fields — fail to replicate is documented through large-scale, preregistered, peer-reviewed meta-research, not anecdote. The Open Science Collaboration’s 2015 project and the 2016 Nature survey are both widely cited precisely because their methods were transparent and their data openly archived.

Contested: What a failed replication actually proves is debated. A failure could mean the original finding was a false positive; it could mean the effect is real but highly sensitive to context, population, or timing in ways that make exact replication difficult; or, less often, it could mean the replication itself was flawed. Researchers also disagree about how much of the crisis reflects fixable methodological habits versus the genuine complexity of studying variable human behavior across different settings and samples.

Key Takeaways

• The replication crisis is the documented pattern of published, peer-reviewed findings failing to produce the same result when independently repeated — a measured phenomenon, not a vague complaint.
• The landmark 2015 Open Science Collaboration project found only 36% of 100 psychology studies replicated with statistical significance in the original direction, with replicated effect sizes roughly half the original average.
• A 2016 Nature survey of more than 1,500 researchers across fields found 70% had failed to reproduce another scientist’s work and half had failed to reproduce their own.
• Publication bias, flexible post-hoc analysis choices (“p-hacking”), and underpowered studies are the leading explanations for why weak effects get published and then don’t hold up.
• Replication success in the 2015 study tracked more closely with the statistical strength of the original finding than with the replicating team’s skill — pointing to a problem in what gets published, not in how it’s later checked.
• Reforms like preregistration and registered reports are specifically designed to remove the incentive to quietly adjust an analysis until it produces a publishable result.
• A failed replication does not automatically prove the original claim was false; it can also reflect a context-sensitive effect or, less commonly, a flawed replication attempt.

Frequently Asked Questions

Does a failed replication mean the original study was wrong?
Not necessarily. It means the effect didn’t show up again under similar conditions, which can reflect a false original finding, a real effect that depends heavily on context, or, less commonly, problems with the replication itself. Researchers weigh all three possibilities rather than assuming the first automatically.

Is this just a psychology problem?
Psychology is where the crisis has been most rigorously measured, largely because psychologists led the large replication projects that quantified it. But the 2016 Nature survey found self-reported replication failures across many scientific disciplines, suggesting the underlying pressures — publication bias, pressure to publish, low statistical power — are not unique to one field.

Does this mean I shouldn’t trust scientific findings?
It means a single study, especially one with a surprising or dramatic result, deserves caution until it has been replicated or supported by a body of converging evidence. It does not mean the scientific process as a whole is broken — if anything, the fact that the crisis itself was uncovered, measured, and published is evidence that science’s self-correcting mechanisms are working.

What is “p-hacking”?
It’s the practice, usually unintentional, of making a series of small analytical decisions after seeing preliminary results — which data to exclude, which subgroup to focus on, when to stop collecting data — that increase the odds of landing on a statistically significant finding, even when the underlying effect is weak or nonexistent.

What are registered reports, and how do they help?
A registered report is a study format in which a journal reviews and conditionally accepts the research question and methodology before the results are known. Because acceptance doesn’t depend on the outcome, it removes the incentive to bury a null result or quietly tweak the analysis to find a significant one.

Has anything actually improved since the 2015 study?
Preregistration and registered reports have become standard options at a growing number of journals, and large-scale replication projects are now a recognized part of the research landscape rather than a novelty. Whether these reforms have measurably raised overall replication rates across science is still being studied.

The Bottom Line

The replication crisis is uncomfortable precisely because it was uncovered by the same rigorous methods science uses to make any other discovery. That is also the reason to take it seriously rather than conclude the scientific enterprise can’t be trusted: the system caught its own weak spot, measured it in public, and has spent a decade building tools — preregistration, registered reports, large collaborative replication projects — specifically designed to shore it up. The practical lesson for anyone reading science news is simpler than the statistics behind it: one study, however striking, is a data point, not a verdict. Treat it that way until it’s been checked.