Should confidence intervals take interpoll variation into account?

An interesting fact I learned from Charles Franklin’s talk is that the variance among polls taken on the same day concerning the same election is about 50% greater than what you’d expect from sampling error alone; that’s because different poles use different sampling methodologies, different question phrasings, different likely-voter weightings, etc.

But when polls report a margin of error, they’re using a 95% confidence interval based on the sampling variance alone.  Should they instead be reporting larger error bars, based on what we know empirically about the extent to which poll results vary from the ground truth?

Tagged , ,

One thought on “Should confidence intervals take interpoll variation into account?

  1. Ben Wieland says:

    Robin Hanson said something like that here. In the body of the post, he quotes Henrion and Fischhoff that 7% of publications of fundamental constants of physics lie outside their 98% confidence intervals. In the comments he comes pretty close to advocating adjusting confidence intervals by historic standards. It seems absurd to hold political polls to higher standards than physics, but let’s start in physics.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 552 other followers

%d bloggers like this: