Should we fire people who pick the wrong Final Four?

A thought experiment touched off by Cathy’s latest post on value-added modeling.

Suppose I’m in charge of a big financial firm and I made every trader who worked for me fill out an NCAA tournament bracket.  Then, every year, I fired the people whose brackets ended up in the lowest quintile.

This makes sense, right?  Successful prediction of college basketball games involves a lot of the same skills you want traders to have:  an ability to aggregate information about uncertain outcomes, fluency in quantitative reasoning, a certain degree of strategic thinking (what choices do you make if your objective is to minimize the probability that your bracket is in the bottom 20%?  What if your fellow traders are also following the same strategy…?)  You might even do a study that finds that firms whose traders did better at bracket prediction actually ended up with better returns 5 years later.  Even if the effect is small, that might add up to a lot of money.  Yes, the measure isn’t perfect, but why wouldn’t I want to fire the people who, on average, are likely to make less money for my firm?

And yet we wouldn’t do this, right?  Just because we think it would be obnoxious to fire people based on a measure predominantly not under their control.  At least we think this when it comes to high-paid financial professionals.  Somehow, when it comes to schoolteachers, we think about it differently.

Tagged , , ,

8 thoughts on “Should we fire people who pick the wrong Final Four?

  1. JSE says:

    I am, of course, aware that financial people ALREADY, in a sense, get fired for scores that are largely independent of their actions: namely, the performance of their portfolios.

  2. Eugene says:

    Would your intuition be any different if we were talking about hiring people rather than firing them? Because in hiring, the kind of thing you’re talking about already happens, and under the same cover of being “data-driven.” Your example actually doesn’t seem all that far-fetched to me.

  3. Craig says:

    When testing school children did not easily lead to firing the “bad teachers”, Wisconsin politicians adopted a new tactic: test to identify the lower-performing “bad” schools and force them to become charter schools! This is the latest here. (An intelligent person might ask whether the tests were designed and validated to serve those purposes) In the long run, though, the politicians may succeed in closing the public school system by making teaching such a thankless profession that college students will not want to become teachers.

  4. Joshua says:

    Jordan, while I agree with the ultimate point that teacher assessment and management through value-added models is bad, this analogy is very weak. People in finance, like people in almost all other industries, mostly get fired for reasons that are outside of their control.

  5. haishan says:

    I will bite this bullet! I think that you should, indeed, fire people who can’t predict the tournament well — especially if there’s quantitative evidence that performance on the brackets is predictive of returns down the line. It’s probably smarter to base decisions on average performance over several years, because the measure is so noisy, but that’s a nitpick.

    Like Eugene says, we already use proxies for ability in hiring — not just in finance or teaching, but in pretty much everything, especially when we’re talking about entry-level positions. And this is pretty clearly a good thing; should employers not be allowed to look at things like what school you attended? Should Google be disallowed from asking programming problems in interviews? (After all, actually working at Google involves doing work that’s pretty substantially different from those problems, even if they load on many of the same skills.) I don’t see why using additional, less-precise data to improve personnel decisions should stop on the first day of work.

  6. Eugene says:

    Haishan, using proxies in hiring is not a good thing or a bad thing in itself. Everything depends on the quality of the proxy and how you use it. Bad ways of using proxies include forgetting that it’s just a proxy and picking an irrelevant proxy.

    What typically happens in hiring decisions is that you start with a weak system for measuring performance (one common failing: being way too tied to the status quo). Then you pick a proxy that looks like it might be a little bit correlated with that measure. Finally, you base your decisions on the proxy alone. The problem with firing people based on VAM scores, or tournament predictions, is that it’s bad management: lazy, possibly biased, and resulting in too many bad decisions. As an employee, I wouldn’t accept a measure that’s just a tiny bit better than random, and neither should you. Good managers need to do the work to find better, more convincing measures of performance and predictors of performance. That’s what they’re supposed to be getting paid for.

  7. haishan says:

    Give me enough barely-better-than-random measures and a place to stand and I can move the world.

    On the object-level point, I’m basically in agreement: the people using teacher VAM don’t understand its extensive limitations, and therefore are using it in ways that are destructive. It may even be that VAM has such poor validity that it’s unfixable. But in general, I think proxies help, and can help a lot. (Yes, even if they’re biased and result in some bad decisions. It’s a question of how to make fewer bad decisions.) They just shouldn’t be used for decision-making in isolation. Perhaps this is a “why oh why can’t we have better managerial science” lament more than anything, but I’d ask how we can use VAM or NCAA tournament forecasting or whatever better, rather than just write PROXIES BAD on a stone tablet and send it down from the mountain.

  8. jessemckeown says:

    Shall we iterate and fire the executives whose hires are in the bottom quintile?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: