MALBEC: Jerry Zhu, Michael Coen, how to say snake in gibbon

Jerry Zhu will give the  last MALBEC seminar of the year tomorrow (Wednesday) afternoon, at 4pm, in Van Vleck B102:

Jerry Zhu (UW, computer sciences)

HAMLET (Human, Animal, and Machine Learning: Experiment and Theory)

Machine learning studies the principles governing all learning systems. Human beings and animals are learning systems too, and can be explored using the same mathematical tools.  This approach has been fruitful in the last few decades with standard tools such as reinforcement learning, artificial neural networks, and non-parametric Bayesian statistics.  We bring the approach one step further with some latest tools in machine learning, and uncover new quantitative findings.  In this talk, I will present three examples: (1) Human semi-supervised learning. Consider a child learning animal names.  Dad occasionally points to an animal and says “Dog!” (labeled data). But mostly the child observes the world by herself without explicit feedback (unlabeled data).  We show that humans learn from both labeled and unlabeled data, and that a simple Gaussian Mixture Model trained using the EM algorithm provides a nice fit to human behaviors.  (2) Human active learning.  The child may ask “What’s that?”, i.e. actively selecting items to query the target labels.  We show that humans are able to perform good active learning, achieving fast exponential error convergence as predicted by machine learning theory.  In contrast, when passively given i.i.d. training data humans learn much slower (polynomial convergence), also predicted by learning theory.  (3) Monkey online learning.  Rhesus monkeys can learn a “target concept”, in the form of a certain shape or color.  What if the target concept keeps changing?  Adversarial online learning model provides a polynomial mistake bound.  Although monkeys perform worse than theory, anecdotal evidence suggests that they follow the concepts better than some graduate students. Finally, I will speculate on a few lessons learned in order to create better machine learning algorithms.

In the third MALBEC lecture, Michael Coen talked about his work on clustering; he asked me afterwards whether I thought the talk was “mathy enough” for the audience, which was funny, because I thought it was 100% math from start to finish!  Here’s a cartoon of the main idea.  When presented with a giant set of data points, one of the first things you might want to do is cluster it:  that is, partition the points into some disjoint collection of subsets, each one of which consists of points which resemble their clustermates more than they do the points in the other clusters.  You might, for instance, want to identify clusters among U.S. legislators, or images, or gene expression patterns. As is so often the case, Cosma Shalizi supplies a good, succinct introduction to the topic from a statistician’s perspective.

How do you know when your clustering algorithm is good?  Sometimes there’s a natural way to evaluate; if your algorithm for clustering legislators reliably separates Democrats from Republicans, you’re probably doing something right.  But with other data, you might not have any pre-existing classification that helps you gauge the reasonableness of your clustering.  Let’s say, for instance, you have lots of short recordings of a gibbon; maybe you think that rather than being scattered structurelessly around the space of 1-second sound clips, they fall into a small finite set of clusters, which you would naturally be tempted to call phonemes. You can run a clustering algorithm on the clips, and you’ll get an answer.  But is it meaningful?  It’s hard to tell without a population of clips which are classified in advance.  Unfortunately, there’s no corpus of written gibbon texts which you can ask gibbons to read aloud.  So you have to do something else.

The trick, as Coen observes, is to replace the difficult and not-very-well-defined question “Is clustering X good?” with the much more tractable question “Are clusterings X and Y similar?”  Coen presented a really nice, principled way of answering this latter question, which allows him to do something like the following:  given your set of audio clips, apply your clustering algorithm separately  to two random samples of 50% of the data points.  These two samples will overlap in around 25% of the data.  Now you can use Coen’s measure to compare the two clusterings induced on this 25% subsample.  If you do this a lot, and you always get two clusterings which are almost exactly the same in Coen’s sense, that’s a good sign that your clustering algorithm is actually capturing real features of the data.

So it turns out that gibbon utterances really do seem to be organized into phonemes.  (A cursory google search suggests that this contradicts conventional wisdom about primate vocalization — can any primatologists weigh in?)  Once you have this finding, and the ability to classify the sound clips, you can do some remarkable things:  for instance, you can look at what combinations of phonemes gibbons emit when a snake comes by.  It turns out that the vocalization elicited by a snake isn’t a consistent combination of phonemes, as it would be in a human language.  Rather, you can write down a finite state automaton, any one of whose outputs seems to be a legitimate gibbon word for “snake”!

Coen had a picture of the automaton on a slide, which is truly cool, but which he is keeping to himself until the paper’s published.  I promise to tell you exactly how to say “snake” in gibbon in a later post.

More MALBEC: Niyogi on geometry of data, Coen on abstract nonsense

Tuesday, April 21 — tomorrow! — brings the third lecture in the MALBEC series:  Michael Coen, of computer sciences and biostat, talks on “Toward Formalizing “Abstract Nonsense”,” in Computer Sciences 1221 at 4pm.  Here’s the abstract:

The idea of a category — a set of objects sharing common properties
— is a fundamental concept in many fields, including mathematics,
artificial intelligence, and cognitive and neuroscience.  Numerous
frameworks, for example, in machine learning and linguistics, rest
upon the simple presumption that categories are well-defined.  This is
slightly worrisome, as the many attempts formalizing categories have
met with equally many attempts shooting them down.

Instead of approaching this issue head on, I derive a robust theory of
“similarity,” from a biologically-inspired approach to perception in
animals.  The very idea of creating categories assumes some implicit
notion of similarity, but it is rarely examined in isolation.
However, doing so is worthwhile, and I demonstrate the theory’s
applicability to a variety of natural and artificial learning
problems.  Even when faced with Watanabe’s “Ugly Duckling” theorem or
Wolpert’s stingy cafeteria (serving the famous “No Free Lunch”
theorems), one can make significant progress toward formalizing a
theory of categories by examining their often unstated properties.

I demonstrate practical applications of this work in several domains,
including unsupervised machine learning, ensemble clustering, image
segmentation, human acquisition of language, and cognitive
neuroscience.

(Joint work with M.H.Ansari)

Delicious food will follow the talk, as if this delicious abstract isn’t enough!

On Friday,  Partha Niyogi gave a beautiful talk on “Geometry, Perception, and Learning.”  His work fits into a really exciting movement in data analysis that one might call “use the submanifold.”  Namely:  data is often given to you as a set of points in some massively high-dimensional space.  For instance, a set of images from a digital camera can be thought of as a sequence of points in R^N, where N is the number of pixels in your camera, a number in the millions, and the value of the Nth coordinate is the brightness of the Nth pixel.  A guy like Niyogi might want to train a computer to distinguish between pictures of horses and pictures of fish.  Now one way to do this is to try to draw a hyperplane across R^N with all the horses are on one side and all the fish on the other.  But the dimension of the space is so high that this is essentially impossible to do well.

But there’s another way — you can take the view that the N-dimensionality of the space of all images is an illusion, and that the images you might be interested in — for instance, some class of images including all horses and all fish — might lie on some submanifold of drastically smaller dimension.

If you believe that manifold is linear, you’re in business:   statisticians have tons of tools, essentially souped-up versions of linear regression, for fitting a linear subspace to a bunch of data.  But linearity is probably too much to ask for.  If you superimpose a picture of a trout on a picture of a walleye, you don’t get a picture of a fish; which is to say, the space of fish isn’t linear.

So it becomes crucial to figure out things about the mystery “fish manifold” from which all pictures of fish are sampled; what are its connected components, or more generally its homology?  What can we say about its curvature?  How well can we interpolate on it to generate novel fish-pictures from the ones in the input?  The work of Carlsson, Diaconis, Ghrist, etc. that I mentioned here is part of the same project.

And in some sense the work of Candes, Tao, and a million others on compressed sensing (well-explained on Terry’s blog) has a similar flavor.  For Niyogi, you have a bunch of given points in R^N and a mystery manifold which is supposed to contain, or at least be close to, those points.  In compressed sensing, the manifold is known — it’s just a union of low-dimensional linear subspaces parametrizing vectors which are sparse in a suitable basis — but the points are not!