“Homological stability for Hurwitz spaces… II” temporarily withdrawn

Akshay Venkatesh, Craig Westerland and I have temporarily withdrawn our preprint “Homological stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields, II,” because there is a gap in the paper which we do not, at present, see how to remove.  There is no reason to think any of the theorems stated in the paper aren’t true, but because some of them are not proved at this time, we’ve pulled back the whole paper until we finish preparing a revised version consisting just of the material that does in fact follow from the arguments in their current form, together with some patches we’ve come up with.   We are extremely grateful to Oscar Randall-Williams for alerting us to the problem in the paper.

I’ll explain where the gap is below the fold, and which parts of the paper are still OK, but first a few thoughts about the issue of mistakes in mathematics.  Of course we owe a lot of people apologies.  All three of us have given talks in which we told people we had a proof of (a certain version of) the Cohen-Lenstra conjecture over F_q(t).  But we do not.  I know several people who had work in progress using this theorem, and so of course this development disrupts what they were doing, and I’ve kept those people up-to-date with the situation of the paper.  If there are others planning immediately to use the result, I hope this post will help draw their attention to the fact that they need to go back to treating this assertion as a conjecture.

One thing I found, when I talked to colleagues about this situation, is that it’s more common than I thought.  Lots of people have screwed up and said things in public or written things in papers they later realized were wrong.  One senior colleague told me an amazing story — he was in the shower one day when he suddenly realized that a paper he’d published in the Annals four years previously, a result he hadn’t even thought about in months, was wrong; there was an induction argument starting from a false base case!  Fortunately, after some work, he was able to construct a repaired argument getting to the same results, which he published as a separate paper.

Naturally nobody likes to talk about their mistakes, and so it’s easy to get the impression that this kind of error is very rare.  But I’ve learned that it’s not so rare.  And I’m going to try to talk about my own error more than I would in my heart prefer to, because I think we have to face the fact that mathematicians are human, and it’s not safe to be certain something is true because we found it on the arXiv, or even in the Annals.

In a way, what happened with our paper is exactly what people predicted would happen once we lost our inhibitions about treating unrefereed preprints as papers.  We wrote the paper, we made it public, and people cited it before it was refereed, and it was wrong.

But what would have happened in a pre-arXiv world?  The mistake was pretty subtle, resting crucially on the relation between this paper and our previous one.  Would the referee have caught it, when we didn’t?  I’m not so sure.  And if the paper hadn’t been openly shared before publication, Oscar wouldn’t have seen it.  It might well have been published in its incorrect form.  On balance, I’d guess wide distribution on arXiv makes errors less likely to propagate through mathematics, not more.

Sociology of mathematics ends here; those who want to know more about the mistake, keep reading past the fold.

Homological Stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields, II

Akshay Venkatesh, Craig Westerland, and I, recently posted a new paper, “Homological Stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields, II.” The paper is a sequel to our 2009 paper of the same title, except for the “II.”  It’s something we’ve been working on for a long time, and after giving a lot of talks about this material it’s very pleasant to be able to show it to people at last!

The main theorem of the new paper is that a version of the Cohen-Lenstra conjecture over F_q(t) is true.  (See my blog entry about the earlier paper for a short description of Cohen-Lenstra.)

For instance, one can ask: what is the average size of the 5-torsion subgoup of a hyperelliptic curve over F_q? That is, what is the value of

where C ranges over hyperelliptic curves of the form y^2 = f(x), f squarefree of degree n?

We show that, for q large enough and not congruent to 1 mod 5, this limit exists and is equal to 2, exactly as Cohen and Lenstra predict. Our previous paper proved that the lim sup and lim inf existed, but didn’t pin down what they were.

In fact, the Cohen-Lenstra conjectures predict more than just the average size of the group as n gets large; they propose a the isomorphism class of the group settles into a limiting distribution, and they say what this distribution is supposed to be! Another way to say this is that the Cohen-Lenstra conjecture predicts that, for each abelian p-group A, the average number of surjections from to A approaches 1. There are, in a sense, the “moments” of the Cohen-Lenstra distribution on isomorphism classes of finite abelian p-groups.

We prove that this, too, is the case for sufficiently large q not congruent to 1 mod p — but, it must be conceded, the value of “sufficiently large” depends on A. So there is still no q for which all the moments are known to agree with the Cohen-Lenstra predictions. That’s why I call what we prove a “version” of the Cohen-Lenstra conjectures. If you think of the Cohen-Lenstra conjecture as being about moments, we’re almost there — but if you think of it as being about probability distributions, we haven’t started!

Naturally, we prefer the former point of view.

This paper ended up being a little long, so I think I’ll make several blog posts about what’s in there, maybe not all in a row.

Random Dieudonne modules, random p-divisible groups, and random curves over finite fields

Bryden Cais, David Zureick-Brown and I have just posted a new paper,  “Random Dieudonne modules, random p-divisible groups, and random curves over finite fields.”

What’s the main idea?  It actually arose from a question David Bryden asked during Derek Garton‘s speciality exam.  We know by now that there is some insight to be gained about studying p-parts of class groups of number fields (the Cohen-Lenstra problem) by thinking about the analogous problem of studying class groups of function fields over F_l, where F_l has characteristic prime to p.

The question David asked was:  well, what about the p-part of the class group of a function field whose characteristic is equal to p?

That’s a different matter altogether.  The p-divisible group attached to the Jacobian of a curve C in characteristic l doesn’t contain very much information;  more or less it’s just a generalized symplectic matrix of rank 2g(C), defined up to conjugacy, and the Cohen-Lenstra heuristics ask this matrix to behave like a random matrix with respect to various natural statistics.

But p-divisible groups in characteristic p are where the fun is!  For instance, you can ask:

What is the probability that a random curve (resp. random hyperelliptic curve, resp. random plane curve, resp. random abelian variety) over F_q is ordinary?

In my view it’s sort of weird that nobody has asked this before!  But as far as I’ve been able to tell, this is the first time the question has been considered.

We generate lots of data, some of which is very illustrative and some of which is (to us) mysterious.  But data alone is not that useful — much better to have a heuristic model with which we can compare the data.  Setting up such a model is the main task of the paper.  Just as a p-divisible group in characteristic l is decribed by a matrix, a p-divisible group in characteristic p is described by its Dieudonné module;  this is just another linear-algebraic gadget, albeit a little more complicated than a matrix.  But it turns out there is a natural “uniform distribution” on isomorphism classes of  Dieudonné modules; we define this, work out its properties, and see what it would say about curves if indeed their Dieudonné modules were “random” in the sense of being drawn from this distribution.

To some extent, the resulting heuristics agree with data.  But in other cases, they don’t.  For instance:  the probability that a hyperelliptic curve of large genus over F_3 is ordinary appears in practice to be very close to 2/3.  But the probability that a smooth plane curve of large genus over F_3 is ordinary seems to be converging to the probability that a random Dieudonné module over F_3 is ordinary, which is

(1-1/3)(1-1/3^3)(1-1/3^5)….. = 0.639….

Why?  What makes hyperelliptic curves over F_3 more often ordinary than their plane curve counterparts?

(Note that the probability of ordinarity, which makes good sense for those who already know Dieudonné modules well, is just the probability that two random maximal isotropic subspaces of a symplectic space over F_q are disjoint.  So some of the computations here are in some sense the “symplectic case” of what Poonen and Rains computed in the orthogonal case.

We compute lots more stuff (distribution of a-numbers, distribution of p-coranks, etc.) and decline to compute a lot more (distribution of Newton polygon, final type…)  Many interesting questions remain!

Poonen-Rains and lines on a quadric surface

One feature of the Poonen-Rains heuristics that might seem strange at first is that the dimension of the Selmer group isn’t 0 almost all  the time.  This is by contrast with the Cohen-Lenstra heuristic, where the p-torsion in the class group is indeed trivial about 1-1/p of the time.  Instead, the Poonen-Rains heuristics predict that the p-Selmer rank is 0 about half the time and 1 about half the time, with about 1/p’s worth of measure devoted to ranks 2 or higher.  Of course, given that we expect a random elliptic curve to have Mordell-Weil rank 1 half the time, it would be bad news for their heuristic if it predicted a lower frequency of positive Selmer rank!

But why is the intersection of two maximal isotropic subspaces 1 half the time and 0 half the time?  You can get a nice picture of what’s going on by thinking about the case of  a quadratic form Q in 4 variables.  The vanishing of the quadratic form cuts out a quadric surface in P^3.  A maximal isotropic subspace is a 2-dimensional space on which Q vanishes — in other words, a line on the quadric.  The intersection of two maximal isotropics is o-dimensional if the corresponding lines are disjoint, 1-dimensional if the lines intersect at a point, and 2-dimensional when the lines coincide.  So what’s the probability that two random lines on the quadric intersect?  The key point is that there are two families of lines.  If L1 and L2 come from different families, they intersect; if they come from the same family, they’re disjoint (except in the unlikely event they coincide.)  So there you go — the intersection of the maximal isotropics is split 50-50 between 0-dimensional and 1-dimensional.  More generally, the variety of maximal isotropic subspaces in an even-dimensional orthogonal space has two components, and this explains the leading term of Poonen-Rains.

It would be interesting to understand how to describe the “two types of maximal isotropics” in the infinite-dimensional F_p-vector space considered by Poonen-Rains, and to understand why the two maximal isotropics supplied by a given elliptic curve lie in the same family if and only if the L-function of E has even functional equation, which should lead one to expect that Sel_p(E) has even rank  (or even, thanks to recent progress on the parity conjecture by Nekovar, Kim, los Dokchitsers, etc., implies that Sel_p(E) has even rank, subject to finiteness of Sha.)

 

 

 

 

Poonen-Rains, Selmer groups, random maximal isotropics, random orthogonal matrices

At the AIM workshop on Cohen-Lenstra heuristics last week I got to hear Bjorn Poonen give a terrific talk about his recent work with Eric Rains about the distribution of mod p Selmer groups in a quadratic twist family of elliptic curves.

Executive summary:  if E is an elliptic curve, say in Weierstrass form y^2 = f(x), and d is a squarefree integer, then we can study the mod p Selmer group Sel_d(E) of the quadratic twist dy^2 = f(x), which sits inside the Galois cohomology H^1(G_Q, E_d[p]).  This is a finite-dimensional vector space over F_p.  And by analogy with the Cohen-Lenstra heuristics for class groups, we can ask whether these groups obey a probability distribution as d varies — that is, does

Pr(dim Sel_d(E) = r | d in [-B, … B])

approach a limit P_r as B goes to infinity, and if so, what is it?

The Poonen-Rains heuristic is based on the following charming observation.  The product of the local cohomology groups H_1(G_v, E[p]) is an infinite-dimensional F_p-vector space endowed with a bilinear form coming from cup product.  In here you have two subspaces:  the image of global cohomology, and the image of local Mordell-Weil.  Each one of these, it turns out, is maximal isotropic — and their intersection is exactly the Selmer group.  So the Selmer group can be seen as the intersection of two maximal isotropic subspaces in a very large quadratic space.

Heuristically, one might think of these two subspaces as being randomly selected among maximal isotropic subspaces.  This suggests a question:  if P_{r,N} is the probability that the intersection of two random maximal isotropics in F_p^{2N} has dimension r, does P_{r,N} approach a limit as N goes to infinity?  It does — and the Poonen-Rains heuristic then asks that the probability that dim Sel_d(E)  = r approaches the same limit.  This conjecture agrees with theorems of Heath-Brown, Swinnerton-Dyer, and Kane in the case p=2, and with results of Bhargava and Shankar when p <= 5 (Bhargava and Shankar work with a family of elliptic curves of bounded height, not a quadratic twist family, but it is not crazy to expect the behavior of Selmer to be the same.)  And in combination with Delaunay’s heuristics for variation of Sha, it recovers Goldfeld’s conjecture that elliptic curves are half rank 0 and half rank 1.

Johan de Jong wrote about a similar question, concentrating on the function field case, in his paper “Counting elliptic surfaces over finite fields.”  (This is the first place I know where the conjecture “Sel_p should have size 1+p on average” is formulated.)  He, too, models the Selmer group by a “random linear algebra” construction.  Let g be a random orthogonal matrix over F_p; then de Jong’s model for the Selmer group is coker(g-1).  This is a natural guess in the function field case:  if E is an elliptic curve over a curve C / F_q, then the Selmer group of E is a subquotient of the etale H^2 of an elliptic surface S over F_q; thus it is closely related to the coinvariants of Frobenius acting on the H^2 of S/F_qbar.  This H^2 carries a symmetric intersection pairing, so Frobenius (after scaling by q) is an orthogonal matrix, which we want to think of as “random.”  (As first observed by Friedman and Washington, the Cohen-Lenstra heurstics can be obtained in similar fashion, but the relevant cohomology is H^1 of a curve instead of H^2 of a surface; so the relevant pairing is alternating and the relevant statistics are those of symplectic rather than orthogonal matrices.)

But this presents a question:  why do these apparently different linear algebra constructions yield the same prediction for the distribution of Selmer ranks?

Here’s one answer, though I suspect there’s a slicker one.

A nice way to describe the distributions that arise in problems of Cohen-Lenstra type is by computing their moments.  But the usual moments (e.g. “expected kth power of dimension of Selmer” or “kth power of order of Selmer” tend not to behave so well.  Better is to compute “expected number of injections from F_p^k into Selmer,” which has a cleaner answer in every case I know.  If the size of the Selmer group is X, this number is

(X-1)(X-p)….(X-p^{k-1}).

Evidently, if you know these “moments” for all k, you can compute the usual moments E(X^k) (which are indeed computed explicitly in Poonen-Rains) and vice versa.

Now:  let A be the random variable (valued in abelian groups!)  “intersection of two random maximal isotropics in a 2N-dimensional quadratic space V” and B be “coker(g-1) where g is a random orthogonal N x N matrix.”

The expected number of injections from F_p^k to B is just the number of injections from F_p^k to F_p^N which are fixed by g.  By Burnside’s lemma, this is the number of orbits of the orthogonal group on Inj(F_p^k, F_p^N).  But by Witt’s Theorem, the orbit of an injection f: F_p^k -> F_p^N is precisely determined by the restriction of the orthogonal form to F_p^k; the number of symmetric bilinear forms on F_p^k is p^((1/2)k(k+1)) and so this is the expected value to be computed.

What about the expected number of injections from F_p^k to A?  We can compute this as follows.  There are about p^{Nk} injections from F_p^k to V.  Of these, about p^{2Nk – (1/2)k(k+1)} have isotropic image.  Call the image W;  we need to know how often W lies in the intersection of the two maximal isotropics V_1 and V_2.  The probability that W lies in V_1 is easily seen to be about p^{-Nk + (1/2)k(k+1)}, and the probability that W lies in V_2 is the same; these two events are independent, so the probability that W lies in A is about p^{-2NK + (1/2)k(k+1)}.  Summing over all isotropic injections gives an expected number of p^{(1/2)k(k+1)} injections from F_p^k to A.  Same answer!

(Note:  in the above paragraph, “about” always means “this is the limit as N gets large with k fixed.”)

What’s the advantage of having two different “random matrix” formulations of the heuristic?  The value of the “maximal isotropic intersection” model is clear — as Poonen and Rains show, the Selmer really is an intersection of maximal isotropic subspaces in a quadratic space.  One value of the “orthogonal cokernel” model is that it’s clear what it says about the Selmer group mod p^k.

Question: What does the orthogonal cokernel model predict about the mod-4 Selmer group of a random elliptic curve?  Does this agree with the theorem of Bhargava and Shankar, which gives the first moment of Sel_4 in a family of elliptic curves ordered by height?

Maples on Cohen-Lenstra for matrices with iid entries

Emmanuel Kowalski pointed me to a very interesting recent paper by Kenneth Maples, a grad student at UCLA working under Terry Tao.  One heuristic justification for the Cohen-Lenstra conjectures, due to Friedman and Washington, relies on the remarkable fact that if M is a random nxn matrix in M_n(Z_p), the distribution of coker(M) among finite abelian p-groups approaches a limit as n goes to infinity; so it makes sense to talk about “the cokernel of a large random matrix” without specifying the size.  (There’s a fuller discussion of Friedman-Washington in this old post.)

Maples shows that the requirement that M is random — that is, that the entries of M are independently drawn from Z_p with additive Haar measure —  is much stronger than necessary.  In fact, he shows that when the entries of M are drawn independently from any distribution on Z_p satisfying a mild non-degeneracy condition, the distribution of coker(M) converges to the so-called Cohen-Lenstra distribution, as in Friedman-Washington.  That’s pretty cool!  I don’t know any arithmetic circumstance that would naturally produce exotic distributions of this kind, but the result helps bolster one’s psychological sense that the Cohen-Lenstra distribution provides the only sensible notion of “cokernel of random matrix,” in some robust sense.

Universality of random matrix laws is a very active and fast-moving topic, but Maples’ result is the first universality result for p-adic matrices that I know of.  More generally, I think there’s a lot to be gained by understanding how well the richly developed theory of random large real and complex matrices carries over to the p-adic case.

Cohen-Lenstra as categorification

Followup on my earlier claim to have been categorified.

The Cohen-Lenstra conjectures govern the p-adic variation of class numbers of quadratic imaginary fields.  At first glance they look very strange.  If you ask the conjectures what proportion of quadratic imaginary class numbers are indivisible by 3, you don’t get 2/3, as you might expect if “class numbers were random numbers,” but rather the infinite product

(1-1/3)(1-1/9)(1-1/27)….

It gets worse — if you ask for the probability that the class number is indivisible by 27, you get the same infinite product multiplied by some completely meaningless-looking rational function in 1/3.

And you ask:  how could anyone ever come up with these conjectures?

The answer, of course, is that they didn’t think about class numbers at all — they thought about class groups.  And in that language, their conjecture is clean to the point of being self-explanatory:  each finite abelian p-group G appears as the p-primary part of a class group with probability inversely proportional to |Aut(G)|.

Numbers are great; but in this context they are merely the Grothendieck K_0 of the category of finite abelian groups.  If life gives you finite abelian groups, use them.  To pass to the Grothendieck group is to decategorify.  And to decategorify is to tempt fate.

 

 

 

 

 

Homological stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields

Now I’ll say a little bit about the actual problem treated by the new paper with Venkatesh and Westerland.  It’s very satisfying to have an actual theorem of this kind:  for years now we’ve been going around saying “it seems like asymptotic conjectures in analytic number theory should have a geometric reflection as theorems about stable cohomology of moduli spaces,” but for quite a while it was unclear we’d ever be able to prove something on the geometric side.

The new paper starts with the question: what do ideal class groups of number fields tend to look like?

That’s a bit vague, so let’s pin it down:  if you write down the ideal class group of the quadratic imaginary number fields , as d ranges over squarefree integers in [0..X],  you get a list of about finite abelian groups.

The ideal class group is the one of the most basic objects of algebraic number theory; but we don’t know much about this list of groups!  Their orders are more or less under control, thanks to the analytic class number formula.  But their structure is really mysterious.

Random pro-p groups, braid groups, and random tame Galois groups

I’ve posted a new paper with Nigel Boston, “Random pro-p groups, braid groups, and random tame Galois groups.”

The paper proposes a kind of “non-abelian Cohen-Lenstra heuristic.”   A typical prediction:  if S is a randomly chosen pair of primes, each of which is congruent to 5 mod 8, and G_S(p) is the Galois group of the maximal pro-2 extension of Q unramified away from S, then G_S(p) is infinite 1/16 of the time.

The usual Cohen-Lenstra conjectures — well, there are a lot of them, but the simplest one asks:  given an odd prime p and a finite abelian p-group A, what is the probability P(A) that a randomly chosen quadratic imaginary field K has a class group whose p-primary part is isomorphic to A?  (Note that the existence of P(A) — which we take to be a limit in X of the corresponding probability as K ranges over quadratic imaginary fields of discriminant at most X — is not at all obvious, and in fact is not known for any p!)

Cohen and Lenstra offered a beautiful conjectural answer to that question:  they suggested that the p-parts of class groups were uniformly distributed among finite abelian p-groups.  And remember — that means that P(A) should be proportional to 1/|Aut(A)|.  (See the end of this post for more on uniform distribution in this categorical setting.)

Later, Friedman and Washington observed that the Cohen-Lenstra conjectures could be arrived at by another means:  if you take K to be the function field of a random hyperelliptic curve X over a finite field instead of a random quadratic imaginary field, then the finite abelian p-group you’re after is just the cokernel of F-1, where F is the matrix corresponding to the action of Frobenius on T_p Jac(X).  If you take the view that F should be a “random” matrix, then you are led to the following question:

Let F be a random element of GL_N(Z_p) in Haar measure:  what is the probability that coker(F-1) is isomorphic to A?

And this probability, it turns out, is precisely the P(A) conjectured by Cohen-Lenstra.

(But now you cry out:  but Frobenius isn’t just any old matrix!  It’s in the generalized symplectic group!  Yes — and Jeff Achter has shown that, at least as far as the probability distribution on A/pA goes, the “right” random matrix model gives you the same answer as the Friedman-Washington quick and dirty model.  Phew.)

Now, in place of a random quadratic imaginary field, pick a prime p and a random set S of g primes, each of which is 1 mod p.  As above, let G_S(p) be the Galois group of the maximal pro-p extension of Q unramified away from S; this is a pro-p group of rank g. What can we say about the probability distribution on G_S(p)?  That is, if G is some pro-p group, can we compute the probability that G_S(p) is isomorphic to G?

Again, there are two approaches.  We could ask that G_S(p) be a “random pro-p group of rank g.”  But this isn’t quite right; G_S(p) has extra structure, imparted to it by the images in G_S(p) of tame inertia at the primes of S.  We define a notion of “pro-p group with inertia data,” and for each pro-p GWID G we guess that the probability that G_S(p) = G is proportional to 1/Aut(G); where Aut(G) refers to the automorphisms of G as GWID, of course.

On the other hand, you could ask what would happen in the function field case if the action of Frobenius on — well, not the Tate module of the Jacobian anymore, but the full pro-p geometric fundamental group of the curve — is “as random as possible.”  (In this case, the group from which Frobenius is drawn isn’t a p-adic symplectic group but Ihara’s “pro-p braid group.”)

And the happy conclusion is that, just as in the Cohen-Lenstra setting, these two heuristic arguments yield the same prediction.  For the relatively few pro-p groups G such that we can compute Pr(G_S(p) = G), our heuristic gives the right answer.  For several more, it gives an answer that seems consistent with numerical experiments.

Maybe it’s correct!