Tag Archives: iran election

Strategic Vision done in by the digits?

Nate Silver at 538 looks at the trailing digits of about 5000 poll results from secretive polling outfit Strategic Vision, finds a badly non-uniform distribution, and says this strongly suggests that SV is making up numbers.  I’m a fan of Nate’s stuff, both sabermetric and electoral, but I’m not so sure he’s right on this.

Nate’s argument is similar to that of Beber and Scacco’s article on the fraudulence of Iran’s election returns.  Humans are bad at picking “random” numbers; so the last digits of human-chosen (i.e. fake) numbers will look less uniform than truly random digits would.

There are at least three ways Nate’s case is weaker than Beber and Scacco’s.

  1. In the Iranian numbers, there were too many numbers ending in 7 and too few ending in 0, consistent with the empirical finding that people trying to generate random numbers tend to disfavor “round” numbers like those ending in 0 and 5.  The digits from Strategic Vision have a lot of 7s, but even more 8s, and the 0s and 5s are approximately where they should be.
  2. It’s not so clear to me that the “right” distribution for these digits is uniform.  Lots of 7s and 8s, few 1s; maybe that’s because in close polls with a small proportion of undecideds, you’ll see a lot of 48-47 results and not so many 51-41s.  I don’t really know what the expected distribution of the digits is — but the fact that I don’t know is a big clothespin between my nose and any assertion of a fishy smell.
  3. And of course my prior for “major US polling firm invents data out of whole cloth” is way lower than my prior for the Iranian federal government doing the same thing.  Strategic Vision could run up exactly the same numbers that Beber and Scacco found, and you’d still be correct to trust them more than the Iran election bureau.  Unless your priors are very different from mine.

So I wouldn’t say, as Nate does, that the numbers compiled at 538  “suggest, perhaps strongly, the possibility of fraud.”

Update (27 Sep): More from Nate on the Strategic Vision digits.  Here he directly compares the digits from Strategic Vision to digits gathered by the same protocol from Quinnipiac.  To my eye, they certainly look different.  I think this strengthens his case.  If he ran the same procedure for five other national pollsters, and the other five all looked like Quinnipiac, I think we’d be in the position of saying “There is good evidence that there’s a methodological difference between SV and other pollsters which has an effect on the distribution of terminal digits.”  But it’s a long way from there to “The methodological difference is that SV makes stuff up.”

On the other hand, Nate remarks that the deviation of the Quinnipiac digits from uniformity is consistent with Benford’s Law.  This is a terrible thing to remark.  Benford’s law applies to the leading digit, not the last one.   The fact that Nate would even bring it up in this context makes me feel a little shaky about the rest of his computations.

Also, there’s a good post about this on Pollster by Mark Blumenthal, whose priors about polling firms are far more reliable than mine.

Tagged , , , , , , ,

More Iranian election statistics

It’s looking more and more as if the official Iranian election returns were at least partially fictional.  I wrote last week about one unconvincing statistical argument for fraud; now a short paper by Bernd Beber and Alexandra Scacco offers more numbers and makes a stronger case.

Keeping in mind that I like their paper a lot, let me say something about a part of it where I thought a bit more justification was needed.

Consider the following three scenarios for generating 116 digits that are supposed to be random:

  1. Digits produced by 116 spins of a spinner labeled 0,1,…,9.
  2. Final digits of vote totals from 116 Iranian provinces.
  3. Final digits of vote totals from U.S. counties.

Now consider the following possible outcomes:

  • A.  Each digit appears either 11 or 12 times.
  • B. 0 appears only 4% of the time, and the other digits appear roughly 10% of the time.
  • C.  7 appears 17% of the time, 5 appears only 4% of the time, other digits appear roughly 10% of the time.

Which outcome should make you doubt that the digits are truly random?

In scenario 1, I think B and C are suspicious; that level of deviation from the mean is more than you’d expect from random spins.  Outcome B would make you suspect the spinner was biased against landing on 0, and C would make you think the spinner was biased towards 7 and against 5.

But of course, outcome A is much more improbable (or so my mental calculation tells me) than either B or C.  So why does’t it arouse suspicion?  Because there’s no apparent mechanism by which a spinner could be biased to produce near-exactly uniformly distributed results like this.  Your prior degree of belief that the spinner is “fixed” to produce this behavior is thus really low, and so even after observing A your belief in the spinner’s fairness is left essentially unchanged.

In scenario 3, I don’t think any of the three outcomes should raise too much suspicion.  Yes, the probability of seeing deviations from uniformity as large as those in C in random digits is under 5%.  But we have a strong prior belief that U.S. elections aren’t crooked — in this case, I think it’s fair to say that scenarios A,B, and C are all evidence that the digits being faked, but not enough evidence to raise the very small prior to a substantial probablity of fraud.

Scenario 2, the one Beber and Scacco consider, is the most interesting.  Outcome C is the one they found.  In order to estimate the probability of fraud in a Bayesian way, given outcome C, you need three numbers:

  • The probability of seeing outcome C from random digits;
  • The probability of seeing outcome C from digits made up from whole cloth at the ministry;
  • The probability — prior to any knowledge of the election results — that the Iranian government would release false numbers.

The third question isn’t a mathematical one, but let’s stipulate that the answer is substantial — much larger than the analogous probability in the United States.

The first question is the one Beber and Scacco assess in their paper; they get an answer of less than 5%.  That sounds pretty damning — deviations like the “extra 7s” seen in the returns would arise less than 1 in 20 times from authentic election numbers.  In fact, outcomes A,B and C are all pretty unlikely to arise from random digits.

But outcome C is evidence for fraud only if it’s more likely to arise from fake numbers than real ones.  And here we have an interesting question.  Beber and Scacco observe that, in practice, people are bad at choosing random digits; when they try, they tend to pick some numbers more frequently than chance would dictate, and some less.  (Their cites for this include the interesting paper by Philip J. Boland and Kevin Hutchinson, Student selection of random digits, Statistician, 49(4): 519-529, 2000.)

So on these grounds it seems outcome C is indeed good evidence for faked data.  But note that the Boland-Hutchinson data doesn’t just say people are bad at picking random digits — it says they are bad in predictable ways at picking random digits.  Indeed, in each of their four trial groups, participants chose “0” — which just doesn’t “feel random” — between 6.5% and 7.5% of the time, substantially less than the 10% you’d get from a random spinner.

So outcome B, I think, would clearly be evidence for fraud.  But outcome C is a little less cut-and-dried.  Just as it’s not clear what mechanism would make a fixed spinner prone to outcome A, it’s not clear whether it’s reasonable to expect a person trying to pick random numbers to choose lots of numbers ending in “7”.  In Boland and Hutchinson’s study, that digit came up just about exactly 10% of the time.

Here’s one way to get a little more info; let’s say we believe that people trying to imitate random numbers choose 0 less often than they should.  If the Iranian election digits had an overpopulation of 0, you might take this to be evidence against the made-up number hypothesis.

So I checked — and in fact, only 9 out of the 116 digits from the provincial returns, or 7.7%, are 0.  Point, Beber and Scacco.

In the end, it’ll take people with better knowledge of Iranian domestic politics — that is, people with more reliable priors — to determine what portion of the election numbers are fake.  But Beber and Scacco have convinced me, at least, that the provincial returns they studied are more consistent with made-up numbers than with real ones.

Here’s a post from Andrew Gelman’s blog in which Beber and Scacco explain what their tests reveal about the county-level election data.

Update: A more skeptical take on Beber and Scacco from Zach at Alchemy Today, who also makes the point that in order to get this question right it’s a good idea to think about the way in which people’s attempts to choose random numbers deviate from chance.  I think his description of Beber and Scacco’s reasoning as “bogus” is too strong, but his observation that the penultimate digits of the state totals for Obama and McCain are as badly distributed as the final digits of the Iran numbers is a good reminder to be cautious.

Re-update: Beber remarks on Zach’s criticisms here.

Tagged , , , , , , ,
%d bloggers like this: