An In-Depth Analysis of the Failure of State Polling in 2016 and What it Means for 2020

There has been a tremendous amount of talk for the last four years about the failure of state polling to correctly capture President Trump’s late surge in the polls and ushering him into the White House. It’s something we’ve all read about and something that’s considered conventional wisdom at this point.

But what does “failure” mean, exactly? By how much did polls miscall 2016? And perhaps most importantly for the moment we’re in, what could that mean for 2020?

I decided to analyze the polls from 2016, compare them to the actual results, and then compared both to what polls are saying in 2020. If polls are as off this year as they were in 2020, the effects will be monumental, though the results might not be exactly as you expect.

I looked closely at the most important swing states from 2016 and those that could tip the election this year, so this analysis will only cover them. Those states are: Wisconsin, Michigan, Iowa, Ohio, Pennsylvania, Virginia, North Carolina, Georgia, Florida, and Arizona. Most of these states went for Trump in 2016 and are the lynchpin for him to win reelection, or for Biden to clinch his.

Polls in 2016 vs. the Actual Results

First, I wanted to get a baseline understanding of how right or wrong polls were in 2016. I’m not going to really look at the national polls from 2016, as those turned out to be fairly accurate. For example, RealClearPolitics’s average of national polls the day before the election was a 3.2% Clinton lead; in reality, she won the national vote by 2.1%. That’s very good polling, but that’s not where the failure was in 2016.

I’m also emphasizing an average of polls, instead of individual ones. I’ll point out specific polls in the analysis, but I’m mainly focusing on the polls as a whole, since the failure was polls in general, not just specific ones.

So what did the polls say the night before the General Election? Let’s take a look:

Probably the biggest miss from pollsters in 2016 was Wisconsin, a state that Hillary Clinton was so sure she had in the bag that she never bothered to visit the entire campaign. RealClearPolitics had Clinton up 6.5% the night before the election and had even moved the state to “Lean D” (instead of “Toss Up”) in their electoral map. The actual results: Donald Trump won by 0.7%. That’s a swing of 7.2%. It doesn’t take hyperbole to say this was a massive polling error. We’ll keep that in mind for later.

Next up: Michigan. Once again, RCP had the race tilting towards Clinton, but this time as a toss up state. Their average of polls had Clinton winning by 3.4%, but the final results were Trump +0.3%. That’s a miss of 3.7%. Trafalgar Group made itself nationally-known for having Trump up late in Michigan, with a lead of 2%. That means they actually slightly favored Trump more than his actual totals by 1.7%, but again, they were the only ones to call the race correctly.

Iowa was expected to go the President’s way on election night and it did. However, like Michigan and Wisconsin, it was carried by him by much larger numbers than predicted. Polls had Mr. Trump ahead by 3.0%, but in actuality, he carried the state by a massive 9.5%. That’s a miss of 6.5%.

Another state that polls expected to go to President Trump was Ohio, but even still he exceeded expectations. The average of polls had Mr. Trump leading 3.5% before Election Day, with the actual result being a resounding 8.1% victory. That’s another 4.6% miss from polls. Thus far we’ve been averaging a 5.5% miss in favor of the President, which is considerable.

Pennsylvania was another miss for pollsters in 2016, though their failure here was much more modest. Polls had Secretary Clinton winning by 1.9%, but in reality, President Trump carried the state by 0.7%. That’s a 2.6% miss, and the lowest we’ve seen thus far from the swing states we’ve analyzed. Again, Trafalgar Group was the only pollster to have Mr. Trump carrying the state, predicting (very accurately) a 1% victory. Harper polling also had the state as a tie.

Interestingly, Virginia‘s polling not only had Secretary Clinton winning, but was almost perfect. Ms. Clinton actually overperformed in Virginia, a rarity in 2016. RCP had the former Secretary of State up by 5% the night before the election, with her final win at 5.4%. That’s a miniscule polling miss of .4%.

This year, as well as in 2016, North Carolina was getting a great deal of press as a possible swing state. This is another state that polls called correctly, but missed by several percentage points. Polls had the President up just 1.0% the day before the election, but Mr. Trump carried the state by 3.7%. That is a modest miss of 2.7%, once again with President Trump outperforming the polls.

For whatever reason, Georgia was considered by RealClearPolitics to be a toss up, but polls had Mr. Trump consistently in the lead. In fact, polling here turned out to be extremely accurate. The average of polls had President Trump up 4.8%, and he won the state by 5.1%. That’s an extremely accurate average of polls, with a miss of only 0.3%.

While Florida gets a lot of press for being a miss for pollsters in 2016, the average of polls the day before the election actually had President Trump carrying the state. RCP had a nail-biter in Florida, predicting a 0.2% victory for Mr. Trump. The actual total was a 1.2% win for the President. Again, Trafalgar Group predicted a Trump win, but actually had him in the lead by 4%. They were off by 2.8%. The most accurate poll for Florida was actually from CBS/YouGov, which had the race tied.

Last, but not least, is Arizona. Like Virginia, Arizona is an aberration where Secretary Clinton slightly outperformed the polls. The polls had Mr. Trump winning the election by 4.0%, but the final results were 3.5% in his favor. The polls were very accurate here, with a miss of only 0.5%.

So How Off Were Polls in 2016 by State?

Here is a “Too Long, Didn’t Read” breakdown of the polls above, and to summarize:

Swing states where the winner was called correctly: 7

Swing states where the winner was called incorrectly: 3

In the battleground states where the races were called incorrectly, the average polling was off by 4.5%. All of the polls overestimated Secretary Clinton’s support and underestimated President Trump’s.

In the swing states where the races were called correctly, the average polling error was 2.3%. However, if you take out Virginia, Arizona, and Georgia, which were very accurate, the average polling mistake was 3.7%.

Here are the predicted and actual results by state:

Wisconsin: Polls underestimated Mr. Trump’s support by 7.2%

Michigan: Polls underestimated Mr. Trump’s support by 3.7%

Iowa: Polls underestimated Mr. Trump’s support by 6.5%

Ohio: Polls underestimated Mr. Trump’s support by 4.6%

Pennsylvania: Polls underestimated Mr. Trump’s support by 2.6%

Virginia: Polls underestimated Ms. Clinton’s support by 0.4%

North Carolina: Polls underestimated Mr. Trump’s support by 2.7%

Georgia: Polls underestimated Mr. Trump’s support by 0.3%

Florida: Polls underestimated Mr. Trump’s support by 1.0%

Arizona: Polls underestimated Ms. Clinton’s support by 0.5%

What Might this Mean for 2020?

Polls were massively off in several major states in 2016. The three states that were called incorrectly by polls, Wisconsin, Michigan, and Pennsylvania, gave Mr. Trump the Presidency. So the question this year is this: what if the same thing happens again?

Pollsters like to imagine that they’ve corrected their mistakes from 2016 and are now accurately polling the electorate this year. But let’s imagine they haven’t. Let’s say that the massive underrepresentation of Mr. Trump is taking place again and the polling misses from 2016 happen again this year. Below, you’ll see what would happen in 2020 if that is the case, based on current polling.

Here is a list of states with their current polling averages, and what the results would be if pollsters made the same mistakes in that specific state as they did in 2016:

Wisconsin:

Current poll leader: Former Vice President Joe Biden by 6.6%.

Who would win if polls were off by the same amount as 2016? President Trump by 0.6%.

Michigan

Current poll leader: Mr. Biden by 5.2%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 1.5%.

Iowa

Current poll leader: No one. It is an exact tie at the moment.

Who would win if polls were off by the same amount as 2016? President Trump by 6.5%.

Ohio

Current poll leader: Mr. Biden by 3.3%.

Who would win if polls were off by the same amount as 2016? President Trump by 1.3%.

Pennsylvania

Current poll leader: Mr. Biden by 4.3%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 1.7%

Virginia

Current poll leader: Mr. Biden by 11.0%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 11.4%

North Carolina

Current poll leader: Mr. Biden by 0.8%.

Who would win if polls were off by the same amount as 2016? President Trump by 1.9%.

Georgia

Current poll leader: President Trump by 1.0%.

Who would win if polls were off by the same amount as 2016? President Trump by 1.3%.

Florida

Current poll leader: Mr. Biden by 1.3%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 0.3%.

Arizona

Current poll leader: Mr. Biden by 3.2%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 3.7%.

National Polling (Which, of course, doesn’t matter in the Electoral College)

Current poll leader: Mr. Biden by 6.5%.

Who would win if polls were off by the same amount as 2016? Mr. Biden by 5.4%

What Does this All Mean?

Based on this extensive analysis, there are a few overarching conclusions:

First, if there are the same major polling errors as there were in 2016, President Trump would go from “losing” several important swing states to winning them.

Second, accounting for the major polling errors from 2016, President Trump is still facing an uphill battle. Even after giving Mr. Trump the entirety of the polling mistakes from 2016, he still is on pace to lose major states, such as Florida, Arizona, and Pennsylvania.

Third, even correcting for the major polling mistakes from four years ago, Mr. Trump is still behind in the Electoral College. The results, after giving the President over half a dozen points in several states and accounting for a “shy voter effect,” still results in a Biden Electoral College victory of 309-229 (assuming that polls are correct in the other major states, including presuming President Trump carries Texas and Mr. Biden carries New Hampshire).

A Few Things to Point Out about this Analysis

This analysis, of course, is not a representation of what will happen or what I hope will happen. But I thought a study of mistakes in polling from 2016 was important to understand what is happening today.

There is also a pandemic that will heavily influence polling habits that would greatly change who votes, and how those votes are counted.

The results will surely be different than what I wrote above, but if history is a guide, they may turn out fairly similar. It’s also 40 days until election day, three Presidential debates, a VP debate, and news we’ll all be talking about that we didn’t even know was news today. Anything could happen, but I hope you enjoyed reading this analysis and feel more informed about the coming election.