I’m on the hiring committee, I chair the graduate admissions committee, and I’m doing an NSF panel, so basically I’ll be spending much of this month judging and ranking people’s mathematics. There’s a lot I like about these jobs: it’s a very efficient way to get a panorama of what’s going on in math and what people think about it. The actual ranking part I don’t like that much — especially because the nature of hiring, admissions, and grant-making means you’re inevitably putting tons of very worthwhile stuff below the line. I feel like a researcher when I read the proposals, like a bureaucrat when I put scores on them.
But of course the bureaucratic work needs to be done. I’d go so far as to say — if mathematicians aren’t willing to rank each other, others will rank us, and that would be worse.
Quomodocumque 
Counterpoint: we could at least try to imagine a less hierarchical mathematical landscape. What would that look like? In Canada, grants are smaller and more widely distributed, right?
Perhaps pairwise comparisons would feel less bureaucratic. Then the NSF (or whoever needs the ranking) could apply something like this: http://arxiv.org/abs/1512.08949
How many mathematicians on the NSF panels would you say fully realize that math is not really a meritocracy and that their ranking of grant applications is only a vague approximation of the actual “ranking” of the applications (if it exists)? And, I feel like whenever I ask you such questions, you always give a really optimistic answer based on projecting your own feelings and thoughts on other mathematicians. So, after that comment, what’s your answer?
Relatedly, I also thought of Canada and the smaller, more widely-distributed grants there. Given that ranking mathematicians and grants is flawed, how come mathematicians in the US aren’t lobbying for the NSF to do like NSERC does? I mean, I know overcompensating perceived “genius” is the American way, but can’t mathematicians do better?
While it is true that the grants in Canada are smaller and more widely distributed, it doesn’t alleviate the need for evaluation and ranking by the funding agency. Within the population of NSERC funding researchers the amount of funding is differs considerably and these differences can really affect how many graduate students and/or postdocs a P.I. can supervise. Thus there is understandably still a lot of angst over the evaluation process and its many foibles.
I should say that I do prefer the Canadian model (having experienced both) for a variety of reasons. I feel like the funding is more directly related to things that really make a difference in research — namely postdocs and grad students, as oppose to summer salary for the P.I. I guess I feel like the idea of paying summer salary for a researcher to do research in the summer is basically a fiction. Whether or not I landed my NSF grant did not really effect what I did in the summer (but maybe that was just me).
In the Canadian model, I have a large amount of autonomy for postdoc hiring. I can pool my money with one or two other people in our department and hire who we want as a postdoc (without having to lobby some departmental PDF committee to make an offer to my candidate).
The same sort of question came up before- see https://quomodocumque.wordpress.com/2009/01/15/should-nsf-give-more-but-smaller-graduate-fellowships/ As I said then, “When I was last at NSF, this idea was raised in a similar situation but the response of DMS was that it’s a lose because there are bean-counters checking what percentage of submissions get funded and they would note a huge increase in the percentage success of Math if more smaller grants were awarded. This could then give reason for a lowering of the DMS budget, to bring the percentages in line with other areas. This is why institutes are favored – one large grant (satisfies the bean-counters) but payoff to lots of people.”
There’s like 0% chance in our society that the group of people who do well within a given system are more deserving than all those who do not do as well. And I’d say that is also the chance that various demographics are represented equally in the more successful vs less successful groups. So even if you take for granted the idea that scoring applicants is a necessary evil, I think things could be improved by ranking people in more than one dimension and by ending the belief that any of this is meaningful.
Jordan:
Perhaps helpful if you think of yourself as ranking the mathematicians’ work rather than the mathematicians themselves.
Piper: well, if the crucial thing is that there is going to be a cutoff, it does not matter whether you have a cutoff region in a multi-dimensional parameter space or a cutoff value for an R-valued function. It’s not as if in either case we would have made up a Mathematical Goodness Quotient and had started to fetishise it. And how likely is it that all demographics are represented exactly equally in God’s valuation function (“deserving”), if there is one? Is it more likely than for every man-made metric?
valuevar, what? who said the crucial thing was a cutoff and why wouldn’t it matter whether there were other dimensions? fetishise?
in a world without oppression, categorizing people by academic interests should produce representative samples of the population. why not? race is not a thing. gender is barely a thing. i’ve never heard anyone even consider that sexual orientation could matter.
if because of oppression, men succeed more by one metric, and women by another. then yes, changing the metric would have a meaningful effect.
Piper: here, we are getting about who gets grant money on a given year and who doesn’t. Hence, a cutoff.
I agree that race is a social construct, and wasn’t thinking of it particularly. And yes, being in different demographic categories correlates with having different opportunities in life. Still, (a) hypothesising that, given equal economic development across the globe, an end to child malnutrition, and equally strong institutions and similar values and preferences in every country, the proportion of Peruvians (say) among research mathematicians would correspond to their proportion in the world population (more than plausible, I guess), does not imply that (b) I should state that any ranking that results in less than 0.5% Peruvians among the category of “deserving mathematicians”, “top mathematicians”, or “grant recipients” must be changed, or is discriminatory, oppressive or meaningless (or all of the three). I am not implying that you are stating as much – just wanted to clarify my (obvious, I’d say) point.
i do not know what you are about or why you are telling me your point is obvious.
1. “cutoff = crucial”
a cutoff is necessary for grant giving purposes when there are too many potential recipients and not enough money, sure. that necessary cutoff is not the crux of my discontent. you said “well, if the crucial thing is that there is going to be a cutoff, it does not matter whether you have a cutoff region in a multi-dimensional parameter space or a cutoff value for an R-valued function.” which led me to believe by crucial you meant crucial to my concerns. if the biggest problem is a cutoff, it doesn’t matter how one defines the cutoff. okay, the biggest thing isn’t the cutoff. to which you say but there has to be a cutoff! uh-huh.
there has to be a cutoff, but it does matter what that cutoff is, because the most important part isn’t that not everyone is included, but that some people might be disproportionately (systematically) excluded.
2. Peruvians aka Impact vs Intent
if a group of people is defined by a power system and granted access to special favors and this group of people is not (approximately! and also over time) representative of the total population, then there is something not good going on. it either happened already, or if we are in the hypothetically harmonious world, then the not good is coming from that power system, and needs to be fixed. the alternative suggests that it is okay for Peruvians to have more difficulty succeeding for no other reason than that they happened to be born Peruvians (or that perhaps they are inherently less qualified). A system need not explicitly seek to exclude a group to be discriminatory and need changing. (No, having less than .5% does not automatically imply anything. But consistently having less than .01% would be a problem.)
[…] As I mentioned, I’m reading Ph.D. admission files. Each file is read by two committee members and thus each file has two numerical scores. […]