Tag Archives: boston

Music Hack Day

Andrew Bridy, Lalit Jain, Ben Recht and I spent the weekend in Cambridge at Music Hack Day, organized by the Echo Nest and sponsored by just about every company you can think of that cares about both music and technology.  We hacked in a somewhat different spirit than most of the folks there; for us, the Million Song Dataset isn’t a tool for app-building, but a playground where we can test ideas about massive networks and information retrieval.

(Re app-building:  Bohemian Rhapsicord.  Chrome-only.)

We’ve actually been playing with the MSD for a few weeks, and I’ll probably post some of those results later, but let’s start with what we did this weekend.  We wanted to see what aspects of the rules of melody we could find in the dataset.  Which notes like to follow which other notes?  Which chords like to follow which other chords?  If you took piano lessons as a kid you already know the answers to these questions.  Which is kind of the point!  When you start to dig into a giant dataset, the first thing you’d better do is check that it can tell you the things you already know.

We quickly found out that getting a handle on the melodies wasn’t so easy.  The song files in the MSD aren’t transcribed from scores, and they don’t have notes:  there’s pitch data, but it’s in the form of chromata; these keep good track of how the energy of a song segment is distributed across frequency bands, but they don’t necessarily correspond well to notes.  (For instance, what does the chroma of a drum hit sound like?)  We found that only about 2% of the songs in the sample had chromata that were “clean” enough to let us infer notes.

But here’s the good thing about a million — 2% of a million is still a lot!  Actually, to save time, we only analyzed about 100,000 songs — but that still gave us a couple of thousand songs’ worth of chroma to work with.  We threw out all the songs Echo Nest thought were in minor keys, and transposed everything to C.  Then we put all the bigrams, or pairs of successive notes, in a big bag, and computed the frequency of each one in the sample.  And this is what we saw:

Pretty nice, right?  The size of the circle represents the frequency of the note.  C (the tonic) and G (the dominant) are the most common notes, just as they should be.  And the notes that are actually in the C-major scale are noticably more frequent than those that aren’t.  The arrow from note x to note y represents the probability that the note following an x will be y; the thicker and redder the arrow, the greater the transition probability.  These, too, look just as they should.  The biggest red arrow is the one from B to C, which is because a major seventh (correction from commenter: a leading tone) really wants to resolve to tonic.  And the strong “Louie Louie” clique joining C,F, and G is plain to see.

Once you have these numbers, you can start to play around.  Lalit wrote a program that generated notes by random-walking along the graph above: the resulting “song” sounds kind of OK!  You can hear it at the end of our 2-minute presentation:

Once you have this computation, you can do all kinds of fun things.  For example, which songs in the database have the most “unusual” melodies from the point of view of this transition matrix?  It turns out that many of the top scorers are indeed songs whose key Echo Nest has misclassified, or which are in keys (like blues scale) that Echo Nest doesn’t recognize.  There’s also a lot of stuff like this:

Not exactly “Louie Louie.”  Low scorers often sound like this Spiritualized song, with big dynamic shifts but not much tonal stray from the old I-IV-V (and in this case, I think it’s mostly the big red I-V)

A relevant paper:  “Clustering beat-chroma patterns in a large music database,” by Thierry Bertin-Mahieux, Ron Weiss, and Daniel Ellis.

Here I am talking linear algebra with Vladimir Viro, who built the amazing Music N-gram Viewer.

DSC_0179 by thomasbonte, on Flickr

Note our team slogan, a bit hard to read on a slant:  “DO THE STUPIDEST THING FIRST.”

Tagged , , , , , ,

Prickly live on WMBR, 1995 and 1997

Matthew White, formerly of Prickly, now of Chores, sent me these files long ago and I never got around to posting them — two live-in-studio performances by Prickly on WMBR.  The shows cover the majority of their catalogue, with drastically better sound quality than the ancient demo tape I posted previously.  Plus, between-song banter!

Prickly Live on WMBR May 1995

Prickly Live on WMBR Jan 1997

Update:  Above links are long dead but I’ve now put this stuff and more on Google Drive.

Tagged , ,

Random pro-p groups, braid groups, and random tame Galois groups

I’ve posted a new paper with Nigel Boston, “Random pro-p groups, braid groups, and random tame Galois groups.”

The paper proposes a kind of “non-abelian Cohen-Lenstra heuristic.”   A typical prediction:  if S is a randomly chosen pair of primes, each of which is congruent to 5 mod 8, and G_S(p) is the Galois group of the maximal pro-2 extension of Q unramified away from S, then G_S(p) is infinite 1/16 of the time.

The usual Cohen-Lenstra conjectures — well, there are a lot of them, but the simplest one asks:  given an odd prime p and a finite abelian p-group A, what is the probability P(A) that a randomly chosen quadratic imaginary field K has a class group whose p-primary part is isomorphic to A?  (Note that the existence of P(A) — which we take to be a limit in X of the corresponding probability as K ranges over quadratic imaginary fields of discriminant at most X — is not at all obvious, and in fact is not known for any p!)

Cohen and Lenstra offered a beautiful conjectural answer to that question:  they suggested that the p-parts of class groups were uniformly distributed among finite abelian p-groups.  And remember — that means that P(A) should be proportional to 1/|Aut(A)|.  (See the end of this post for more on uniform distribution in this categorical setting.)

Later, Friedman and Washington observed that the Cohen-Lenstra conjectures could be arrived at by another means:  if you take K to be the function field of a random hyperelliptic curve X over a finite field instead of a random quadratic imaginary field, then the finite abelian p-group you’re after is just the cokernel of F-1, where F is the matrix corresponding to the action of Frobenius on T_p Jac(X).  If you take the view that F should be a “random” matrix, then you are led to the following question:

Let F be a random element of GL_N(Z_p) in Haar measure:  what is the probability that coker(F-1) is isomorphic to A?

And this probability, it turns out, is precisely the P(A) conjectured by Cohen-Lenstra.

(But now you cry out:  but Frobenius isn’t just any old matrix!  It’s in the generalized symplectic group!  Yes — and Jeff Achter has shown that, at least as far as the probability distribution on A/pA goes, the “right” random matrix model gives you the same answer as the Friedman-Washington quick and dirty model.  Phew.)

Now, in place of a random quadratic imaginary field, pick a prime p and a random set S of g primes, each of which is 1 mod p.  As above, let G_S(p) be the Galois group of the maximal pro-p extension of Q unramified away from S; this is a pro-p group of rank g. What can we say about the probability distribution on G_S(p)?  That is, if G is some pro-p group, can we compute the probability that G_S(p) is isomorphic to G?

Again, there are two approaches.  We could ask that G_S(p) be a “random pro-p group of rank g.”  But this isn’t quite right; G_S(p) has extra structure, imparted to it by the images in G_S(p) of tame inertia at the primes of S.  We define a notion of “pro-p group with inertia data,” and for each pro-p GWID G we guess that the probability that G_S(p) = G is proportional to 1/Aut(G); where Aut(G) refers to the automorphisms of G as GWID, of course.

On the other hand, you could ask what would happen in the function field case if the action of Frobenius on — well, not the Tate module of the Jacobian anymore, but the full pro-p geometric fundamental group of the curve — is “as random as possible.”  (In this case, the group from which Frobenius is drawn isn’t a p-adic symplectic group but Ihara’s “pro-p braid group.”)

And the happy conclusion is that, just as in the Cohen-Lenstra setting, these two heuristic arguments yield the same prediction.  For the relatively few pro-p groups G such that we can compute Pr(G_S(p) = G), our heuristic gives the right answer.  For several more, it gives an answer that seems consistent with numerical experiments.

Maybe it’s correct!

Tagged , , , , , , , ,
%d bloggers like this: