An interesting fact I learned from Charles Franklin’s talk is that the variance among polls taken on the same day concerning the same election is about 50% greater than what you’d expect from sampling error alone; that’s because different poles use different sampling methodologies, different question phrasings, different likely-voter weightings, etc.

But when polls report a margin of error, they’re using a 95% confidence interval based on the sampling variance alone. Should they instead be reporting larger error bars, based on what we know empirically about the extent to which poll results vary from the ground truth?

### Like this:

Like Loading...

*Related*

Robin Hanson said something like that here. In the body of the post, he quotes Henrion and Fischhoff that 7% of publications of fundamental constants of physics lie outside their 98% confidence intervals. In the comments he comes pretty close to advocating adjusting confidence intervals by historic standards. It seems absurd to hold political polls to higher standards than physics, but let’s start in physics.