How many points does a random curve over F_q have?

So asks a charming preprint by Achter, Erman, Kedlaya, Wood, and Zureick-Brown.  (2/5 Wisconsin, 1/5 ex-Wisconsin!)  The paper, I’m happy to say, is a result of discussions at an AIM workshop on arithmetic statistics I organized with Alina Bucur and Chantal David earlier this year.

Here’s how they think of this.  By a random curve we might mean a curve drawn uniformly from M_g(F_q).  Let X be the number of points on a random curve.  Then the average number of points on a random curve also has a geometric interpretation: it is

$|M_{g,1}(\mathbf{F}_q)|/|M_{g}(\mathbf{F}_q)|$

$|M_{g,2}(\mathbf{F}_q)|/|M_{g}(\mathbf{F}_q)|$?

That’s just the average number of ordered pairs of distinct points on a random curve; the expected value of X(X-1).

If we can compute all these expected values, we have all the moments of X, which should give us a good idea as to its distribution.  Now if life were as easy as possible, the moduli spaces of curves would have no cohomology past degree 0, and by Grothendieck-Lefschetz, the number of points on M_{g,n} would be q^{3g-3+n}.  In that case, we’d have that the expected value of X(X-1)…(X-n) was q^n.  Hey, I know what distribution that is!  It’s Poisson with mean q.

Now M_g does have cohomology past degree 0.  The good news is, thanks to the Madsen-Weiss theorem (née the Mumford conjecture) we know what that cohomology is, at least stably.  Yes, there are a lot of unstable classes, too, but the authors propose that heuristically these shouldn’t contribute anything.  (The point is that the contribution from the unstable range should look like traces of gigantic random unitary matrices, which, I learn from this paper, are bounded with probability 1 — I didn’t know this, actually!)  And you can even make this heuristic into a fact, if you want, by letting q grow pretty quickly relative to g.

So something quite nice happens:  if you apply Grothendieck-Lefschetz (actually, you’d better throw in Kai Behrend’s name, too, because M_g is a Deligne-Mumford stack, not an honest scheme) you find that the moments of X still agree with those of a Poisson distribution!  But the contribution of the tautological cohomology shifts the mean from q to q+1+1/(q-1).

This is cool in many directions!

• It satisfies one’s feeling that a “random set,” if it carries no extra structure, should have cardinality obeying a Poisson distribution — the “uniform distribution” on the groupoid of sets.  (Though actually that uniform distribution is Poisson(1); I wonder what tweak is necessary to be able to tune the mean?)
• I once blogged about an interesting result of Bucur and Kedlaya which showed that a random smooth complete intersection curve in P^3 of fixed degree had slightly fewer than q+1 points; in fact, about q+1 – 1/q + o(q^2).  Here the deviation is negative, rather than positive, as the new paper suggests is the case for general curves; what’s going on?
• I have blogged about the question of average number of points on a random curve before.  I’d be very interested to know whether the new heuristic agrees with the answer to the question proposed at the end of that post; if g is a large random matrix in GSp(Z_ell) with algebraic eigenvalues, and which multiplies the symplectic form by q, and you condition on Tr(g^k) > (-q^k-1) so that the “curve” has nonnegatively many points over each extension of F_q, does this give something like the distribution the five authors predict for Tr(g)?  (Note:  I don’t think this question is exactly well-formed as stated.)

Silas Johnson on weighted discriminants with mass formulas

My Ph.D. student Silas Johnson just posted his thesis paper to the arXiv, and it’s cool, and I’m going to blog about it!

How should you count number fields?  The most natural way is by discriminant; you count all degree-n number fields K with a given Galois group G in S_n and discriminant bounded in absolute value by B.  This gives you a value N_G(B) whose asymptotic behavior in B you might want to study.  The classical results and exciting new ones you’ve heard about — Davenport-Heilbron, Bhargava, and all that — generally concern counts of this kind.

But there are reasons to consider other kinds of counts.  I once had a bunch of undergrads investigate the behavior of N_3(X,Y), the number of cubic fields whose discriminant had squarefree part at most X and maximal square divisor at most Y.  This gives a more refined picture of the ramification behavior of the fields.  Asymptotics for this are still unknown!  (I would expect the main term to be on order $X Y^{1/2}$, but I don’t know what the secondary terms should be.)

One nice thing about the discriminant, though, is that it has a mass formula.  In brief:  a map f from Gal(Q_p) to S_n corresponds to a degree-n extension of Q_p, which has a discriminant (a power of p) which we call Disc(f).  Averaging Disc(f)^{-1} over all homomorphisms f gives you a polynomial in p^{-1}, which we call the local mass.  Now here’s the remarkable fact (shown by Bhargava, deriving from a formula of Serre) — there is a universal polynomial P(x) such that the local mass at p is equal to P(p^{-1}) for every P.  This is not hard to show for the tame primes p (you can see this point discussed in Silas’s paper or in the paper by Kedlaya I linked above) but that it holds for the wild primes is rather mysterious and strange.  And it certainly seems to ratify the idea that there’s something especially nice about the discriminant.  What’s more, this polynomial P is not just some random thing; it’s the product over p of P(p^{-1}) that gives the constant in Bhargava’s conjectural asymptotic for the number of number fields for degree n.

But here’s the thing.  If we replace G by a subgroup of S_n, there need not be a universal mass formula anymore.  Kedlaya (and Daniel Gulotta, in the appendix) show lots of examples.  The simplest example is the dihedral group of order 8.

All is not lost, though!  Wood showed in 2008 that you could fix this problem by replacing the discriminant of a D_4-extension with a different invariant.  Namely:  a D_4 quartic field M has a quadratic subextension L.  If you replace Disc(L/Q) with Disc(L/Q) times the norm to Q of Disc(L/M), you get a different invariant of M — an example of what Silas calls a “weighted discriminant” — and when you compute the local mass according to {\em this} invariant, you get a polynomial in p^{-1} again!

So maybe Wood’s modified discriminant, not the usual discriminant, is the “right” way to count dihedral quartics?  Does the product of her local masses give the right asymptotic for the number of D_4 extensions with Woodscriminant at most B?

It’s not at all clear to me how, if at all, you can cook up a modified discriminant for an arbitrary group G that has a universal mass formula.  What Silas shows is that having a mass formula is indeed special; when G is a p-group, there are only finitely many weighted discriminants that have one.  Silas thinks, and so do I, that this is actually true for every finite group G, and that some version of his approach will eventually prove this.  And he classifies all such weighted discriminants for groups of size up to 12; for any individual G, it’s a computation which can be made nicely algorithmic.  Very cool!

Y. Zhao and the Roberts conjecture over function fields

Before the developments of the last few years the only thing that was known about the Cohen-Lenstra conjecture was what had already been known before the Cohen-Lenstra conjecture; namely, that the number of cubic fields of discriminant between -X and X could be expressed as

$\frac{1}{3\zeta(3)} X + o(X)$.

It isn’t hard to go back and forth between the count of cubic fields and the average size of the 3-torsion part of the class group of quadratic fields, which gives the connection with Cohen-Lenstra in its usual form.

Anyway, Datskovsky and Wright showed that the asymptotic above holds (for suitable values of 12) over any global field of characteristic at least 5.  That is:  for such a field K, you let N_K(X) be the number of cubic extensions of K whose discriminant has norm at most X; then

$N_K(X) = c_K \zeta_K(3)^{-1} X + o(X)$

for some explicit rational constant $c_K$.

One interesting feature of this theorem is that, if it weren’t a theorem, you might doubt it was true!  Because the agreement with data is pretty poor.  That’s because the convergence to the Davenport-Heilbronn limit is extremely slow; even if you let your discriminant range up to ten million or so, you still see substantially fewer cubic fields than you’re supposed to.

In 2000, David Roberts massively clarified the situation, formulating a conjectural refinement of the Davenport-Heilbronn theorem motivated by the Shintani zeta functions:

$N_{\mathbf{Q}}(X) = (1/3)\zeta(3)^{-1} X + c X^{5/6} + o(X^{5/6})$

with c an explicit (negative) constant.  The secondary term with an exponent very close to 1 explains the slow convergence to the Davenport-Heilbronn estimate.

The Datskovsky-Wright argument works over an arbitrary global field but, like most arguments that work over both number fields and function fields, it is not very geometric.  I asked my Ph.D. student Yongqiang Zhao, who’s finishing this year, to revisit the question of counting cubic extensions of a function field F_q(t) from a more geometric point of view to see if he could get results towards the Roberts conjecture.  And he did!  Which is what I want to tell you about.

But while Zhao was writing his thesis, there was a big development — the Roberts conjecture was proved.  Not only that — it was proved twice!  Once by Bhargava, Shankar, and Tsimerman, and once by Thorne and Taniguchi, independently, simultaneously, and using very different methods.  It is certainly plausible that these methods can give the Roberts conjecture over function fields, but at the moment, they don’t.

Neither does Zhao, yet — but he’s almost there, getting

$N_K(T) = \zeta_K(3)^{-1} X + O(X^{5/6 + \epsilon})$

for all rational function fields K = F_q(t) of characteristic at least 5.  And his approach illuminates the geometry of the situation in a very beautiful way, which I think sheds light on how things work in the number field case.

Geometrically speaking, to count cubic extensions of F_q(t) is to count trigonal curves over F_q.  And the moduli space of trigonal curves has a classical unirational parametrization, which I learned from Mike Roth many years ago:  given a trigonal curve Y, you push forward the structure sheaf along the degree-3 map to P^1, yielding a rank-3 vector bundle on P^1; you mod out by the natural copy of the structure sheaf; and you end up with a rank-2 vector bundle W on P^1, whose projectivization is a rational surface in which Y embeds.  This rational surface is a Hirzebruch surface F_k, where k is an integer determined by the isomorphism class of the vector bundle W.  (This story is the geometric version of the Delone-Fadeev parametrization of cubic rings by binary cubic forms.)

This point of view replaces a problem of counting isomorphism classes of curves (hard!) with a problem of counting divisors in surfaces (not easy, but easier.)  It’s not hard to figure out what linear system on F_k contains Y.  Counting divisors in a linear system is nothing but a dimension count, but you have to be careful — in this problem, you only want to count smooth members.  That’s a substantially more delicate problem.  Counting all the divisors is more or less the problem of counting all cubic rings; that problem, as the number theorists have long known, is much easier than the problem of counting just the maximal orders in cubic fields.

Already, the geometric meaning of the negative secondary term becomes quite clear; it turns out that when k is big enough (i.e. if the Hirzebruch surface is twisty enough) then the corresponding linear system has no smooth, or even irreducible, members!  So what “ought” to be a sum over all k is rudely truncated; and it turns out that the sum over larger k that “should have been there” is on order X^{5/6}.

So how do you count the smooth members of a linear system?  When the linear system is highly ample, this is precisely the subject of Poonen’s well-known “Bertini theorem over finite fields.”  But the trigonal linear systems aren’t like that; they’re only “semi-ample,” because their intersection with the fiber of projection F_k -> P^1 is fixed at 3.  Zhao shows that, just as in Poonen’s case, the probability that a member of such a system is smooth converges to a limit as the linear system gets more complicated; only this limit is computed, not as a product over points P of the probability D is smooth at P, but rather a product over fibers F of the probability that D is smooth along F.  (This same insight, arrived at independently, is central to the paper of Erman and Wood I mentioned last week.)

This alone is enough for Zhao to get a version of Davenport-Heilbronn over F_q(t) with error term O(X^{7/8}), better than anything that was known for number fields prior to last year.  How he gets even closer to Roberts is too involved to go into on the blog, but it’s the best part, and it’s where the algebraic geometry really starts; the main idea is a very careful analysis of what happens when you take a singular curve on a Hirzebruch surface and start carrying out elementary transforms at the singular points, making your curve more smooth but also changing which Hirzebruch surface it’s on!

To what extent is Zhao’s method analogous to the existing proofs of the Roberts conjecture over Q?  I’m not sure; though Zhao, together with the five authors of the two papers I mentioned, spent a week huddling at AIM thinking about this, and they can comment if they want.

I’ll just keep saying what I always say:  if a problem in arithmetic statistics over Q is interesting, there is almost certainly interesting algebraic geometry in the analogous problem over F_q(t), and the algebraic geometry is liable in turn to offer some insights into the original question.

Slides from my JMM talk, “How to Count With Topology”

Back from San Diego, recovering from the redeye.  It was a terrific Joint Math Meetings this year; I saw lots of old friends and great talks, but had to miss a lot of both, too.

A couple of people asked me for the slides of my talk, “How To Count with Topology.”  Here they are, in .pdf:

How To Count With Topology

If you find this stuff interesting, these blog posts give a somewhat more detailed sketch of the papers with Venkatesh and Westerland I talked about.

Homological Stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields, II

Akshay Venkatesh, Craig Westerland, and I, recently posted a new paper, “Homological Stability for Hurwitz spaces and the Cohen-Lenstra conjecture over function fields, II.” The paper is a sequel to our 2009 paper of the same title, except for the “II.”  It’s something we’ve been working on for a long time, and after giving a lot of talks about this material it’s very pleasant to be able to show it to people at last!

The main theorem of the new paper is that a version of the Cohen-Lenstra conjecture over F_q(t) is true.  (See my blog entry about the earlier paper for a short description of Cohen-Lenstra.)

For instance, one can ask: what is the average size of the 5-torsion subgoup of a hyperelliptic curve over F_q? That is, what is the value of

$\lim_{n \rightarrow \infty} \frac{\sum_C |J(C)[5](\mathbf{F}_q)|}{\sum_C 1}$

where C ranges over hyperelliptic curves of the form y^2 = f(x), f squarefree of degree n?

We show that, for q large enough and not congruent to 1 mod 5, this limit exists and is equal to 2, exactly as Cohen and Lenstra predict. Our previous paper proved that the lim sup and lim inf existed, but didn’t pin down what they were.

In fact, the Cohen-Lenstra conjectures predict more than just the average size of the group $J(C)[5](\mathbf{F}_q)$ as n gets large; they propose a the isomorphism class of the group settles into a limiting distribution, and they say what this distribution is supposed to be! Another way to say this is that the Cohen-Lenstra conjecture predicts that, for each abelian p-group A, the average number of surjections from $J(C)(\mathbf{F}_q)$ to A approaches 1. There are, in a sense, the “moments” of the Cohen-Lenstra distribution on isomorphism classes of finite abelian p-groups.

We prove that this, too, is the case for sufficiently large q not congruent to 1 mod p — but, it must be conceded, the value of “sufficiently large” depends on A. So there is still no q for which all the moments are known to agree with the Cohen-Lenstra predictions. That’s why I call what we prove a “version” of the Cohen-Lenstra conjectures. If you think of the Cohen-Lenstra conjecture as being about moments, we’re almost there — but if you think of it as being about probability distributions, we haven’t started!

Naturally, we prefer the former point of view.

This paper ended up being a little long, so I think I’ll make several blog posts about what’s in there, maybe not all in a row.