In defense of Nate Silver and experts

Cathy goes off on Nate Silver today, calling naive his account of well-meaning people saying false things because they’ve made math mistakes.  In Cathy’s view, people say false things because they’re not well-meaning and are trying to screw you — or, sometimes, because they’re well-meaning but their incentives are pointed at something other than accuracy.  Read the whole thing, it’s more complicated than this paraphrase suggests.

Cathy, a fan of and participant in mass movements, takes special exception to Silver saying:

This is neither the time nor the place for mass movements — this is the time for expert opinion. Once the experts (and I’m not one of them) have reached some kind of a consensus about what the best course of action is (and they haven’t yet), then figure out who is impeding that action for political or other disingenuous reasons and tackle them — do whatever you can to remove them from the playing field. But we’re not at that stage yet.

Cathy’s take:

…I have less faith in the experts than Nate Silver: I don’t want to trust the very people who got us into this mess, while benefitting from it, to also be in charge of cleaning it up. And, being part of the Occupy movement, I obviously think that this is the time for mass movements.

From my experience working first in finance at the hedge fund D.E. Shaw during the credit crisis and afterwards at the risk firm Riskmetrics, and my subsequent experience working in the internet advertising space (a wild west of unregulated personal information warehousing and sales) my conclusion is simple: Distrust the experts.

I think Cathy’s distrust is warranted, but I think Silver shares it.  The central concern of his chapter on weather prediction is the vast difference in accuracy between federal hurricane forecasters, whose only job is to get the hurricane track right, and TV meteorologists, whose very different incentive structure leads them to get the weather wrong on purpose.  He’s just as hard on political pundits and their terrible, terrible predictions, which are designed to be interesting, not correct.

Cathy wishes Silver would put more weight on this stuff, and she may be right, but it’s not fair to paint him as a naif who doesn’t know there’s more to life than math.  (For my full take on Silver’s book, see my review in the Globe.)

As for experts:  I think in many or even most cases deferring to people with extensive domain knowledge is a pretty good default.  Maybe this comes from seeing so many preprints by mathematicians, physicists, and economists flushed with confidence that they can do biology, sociology, and literary study (!) better than the biologists, sociologists, or scholars of literature.  Domain knowledge matters.  Marilyn vos Savant’s opinion about Wiles’s proof of Fermat doesn’t matter.

But what do you do with cases like finance, where the only people with deep domain knowledge are the ones whose incentive structure is socially suboptimal?  (Cathy would use saltier language here.)  I guess you have to count on mavericks like Cathy, who’ve developed the domain knowledge by working in the financial industry, but who are now separated from the incentives that bind the insiders.

But why do I trust what Cathy says about finance?

Because she’s an expert.

Is Cathy OK with this?

Tagged , , , , ,

5 thoughts on “In defense of Nate Silver and experts

  1. Eugene says:

    “Why do I trust what Cathy says about finance? Because she’s an expert.” Isn’t it more because *you trust Cathy*? If Cathy and ten other people with as much, or more, finance expertise all told you totally different things about the financial crisis, you’d probably believe Cathy, no?

    “I think in many or even most cases deferring to people with extensive domain knowledge is a pretty good default.” You and Cathy have different intuitions about this because you have different experiences. In math, the experts tend to be pretty reliable, and in finance, well, I think Cathy’s blog post pretty much nails it.

    It looks like Silver is smart enough to recognize that incentives affect how reliable the experts are, but doesn’t fully recognize how perverted the incentives in finance (and in politics, and at the intersection of the two) are. Viewed in that context, his “this is the time for expert opinion” line about the financial crisis starts to sound pretty appalling. Who counts as an expert? Who vets the experts?

    Last word (for now) goes to Laurie Anderson:

    I heard her do a version of this in 2005 or so, before the financial crisis (and before without the verse on the bailout). I thought it was pretty heavy-handed at the time. Now… not so much.

  2. [...] friend Jordan has written a response to yesterday’s post about Nate Silver. He is a major fan of Silver and contends that [...]

  3. [...] Ellenberg wrote the following response on Quomodocumque. It’s republished with his [...]

  4. [...] Also via 3QD, Cathy O’Neil doesn’t care much for Nate Silver’s The Signal and the Noise. Jordan Ellenberg replies. [...]

  5. [...] Tai at the law school wrote a long, amazing Facebook message to me about the question Cathy and I have been pawing at:  when and in what spirit should we be listening to experts?  It was too good [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 548 other followers

%d bloggers like this: