Election polls get it wrong mainly because of who answers them, not because of sampling. The margin of error printed under a result covers only the randomness of drawing a sample. It says nothing about the much larger problem of systematic differences between the people who respond and the people who do not.

The 2020 US election produced the clearest illustration on record, and the investigation into it is unusually candid about what pollsters still cannot explain.

How Big the Miss Actually Was

The American Association for Public Opinion Research convened a task force to evaluate 2020 pre-election polling. For polls conducted in the final two weeks, it found an average error on the margin of 4.5 points in national popular vote polls and 5.1 points in state-level presidential polls.

That was the highest error in 40 years for the national popular vote, and the highest in at least 20 years for state-level estimates.

The error also ran in one direction. The average signed error was too favorable to Biden by 3.9 points nationally and 4.3 points in statewide polls. Errors that run one way are not sampling noise. Sampling noise cancels out.

What the Task Force Ruled Out

The value of the report is in what it eliminated, because most popular explanations did not survive.

Late deciders were not the cause: only 4 percent of voters were undecided in the final two weeks. Failure to weight by education was not the cause: 92 percent of late polls weighted for education. Demographic composition errors, turnout modeling problems among either candidate’s supporters and wrong assumptions about the mix of Election Day and early voters were all examined and set aside.

So was the most popular theory of all. The idea that Trump supporters were reluctant to admit their preference to interviewers, the so-called shy voter, did not hold up across interview modes.

What Is Left Is Differential Nonresponse

The likeliest remaining explanation is that the people who take polls are systematically different from the people who do not, in ways that weighting cannot fix.

If too many Democrats and too few Republicans agree to be surveyed, and that gap persists inside every demographic group a pollster weights on, the adjustment does not correct anything. A pollster who weights to the correct share of, say, non-college voters still has the wrong non-college voters if the ones who answer differ politically from the ones who do not.

The report is blunt about the limits here. It states that it is impossible to identify the precise cause of the polling error without knowing the opinions and demographics of the voters who were not included in the polls. That information does not exist, by definition.

Why the Margin of Error Cannot Warn You

A margin of error measures sampling error: the difference between an estimate from a sample and the value you would get from the whole population.

Everything above sits outside that definition. AAPOR notes that surveys are subject to errors ranging from how questions were designed and asked to how interviews were conducted, and that these cannot be measured, so the precise amount of error in any poll finding can never be known.

Pew’s summary of the 2020 review makes the consequence explicit: the margin of error is not the same as total polling error, and reporting it alone creates the impression that results are far more precise than they actually are.

Two Numbers That Mislead Even When Polls Are Right

Two further traps catch careful readers.

First, the headline margin applies to the whole sample, never to a subgroup. In a survey of 1,000 people, a subgroup of 200 respondents carried a margin of plus or minus 6.9 percentage points. Crosstabs are far shakier than the topline they sit under.

Second, comparing two numbers involves the error in both. A candidate leading 48 to 45 in a poll with a 3-point margin does not have a 3-point lead that is just inside the error. The gap itself is well within the uncertainty of the comparison.

What to Look for in a Poll Now

AAPOR’s main recommendation was transparency rather than a technical fix. Pollsters were urged to explain how numbers are produced and to document their adjustment decisions, including their assumptions about partisan composition and what those assumptions do to the result.

For a reader, that is the useful signal. A poll that publishes its weighting decisions and its assumed partisan mix is showing you the part most likely to be wrong. A poll that publishes only a topline and a margin of error is showing you the part least likely to matter.