Two hours less than 30 days ago, the team of BellKor’s Pragmatic Chaos submitted the first entry to the Netflix Prize to exhibit a 10% improvement in performance over Netflix’s movie-recommendation algorithm. That started the final clock for the competition — whoever’s ahead at 2:42 Eastern time today wins the $1 million prize.
One of the really interesting lessons of the competition is that blendings of many algorithms seem to work better than any single algorithm, even when there’s no principled reason to do the blend. It’s sort of a “wisdom of crowds of computer programs” effect. As you can imagine, once BellKor’s Pragmatic Chaos (itself a blend of algorithms from two teams that have been leading through most of the competition) crossed the 10% threshold, just about everybody else realized their best — probably only — chance was to work together.
As of yesterday afternoon, a team called The Ensemble, made up of — well, I can’t really tell how many previously separate competitors, but a lot — has achieved a 10.09% improvement. BellKor’s Pragmatic Chaos is at 10.08%. A hundredth of a percentage point might determine who gets the million. Wow.
Is BPC sitting on a slightly better algorithm they’re planning to submit at the buzzer for the win? Check the leaderboard later this afternoon to find out.
Update: Double wow. No announcement yet, but it looks like BPC and The Ensemble both made submissions in the last hour of the contest; BGC made it to 10.09% but Ensemble, four minutes before closing time, bumped up to 10.10%.
Re-update: I didn’t read the rules carefully enough. It looks like there’s another dataset (‘test dataset”), distinct from the one generating the public scores (“quiz dataset”) and the final winner will be the program that does best on the test dataset. So the shifts in the lead, exciting as they are, aren’t necessarily relevant to the contest; we’ve got two algorithms which are essentially identically good and it ought to be a coin-flip which one does better in the final judgment.
Quomodocumque 
A 10% improvement is not very significant compared to the low lying fruit that Netflix seems to have missed. For starters, they can include a search engine that allows the “AND” construction rather than treating all combinations of words as “OR”. They could also make it easier for couples to distinguish their individual tastes: my wife wants to watch “Amelie”, “Everything is Illuminated”, or “Out of Africa”, whereas I want to watch “Die Hard”, “2001”, or “Harold and Kumar”.
I brought this issue up with the Netflix folks; they’re aware of it and for whatever reason are pretty confident it’s not messing with their overall ratings engine very much, though of course it might be creating weird recommendations for particular accounts like yours. I for one would totally rent “Amelie Dies Hard” or “Harold and Kumar are Illuminated.”
Turns out that BPC has won, having a better test score than The Ensemble.
If one wants to have separate tastes in movies, it is easy to set up multiple queues — even if you have all your movies come from one queue you can rate the movies separately to get separate recommendations — this is what my wife and I do.
[…] Seems that the winning condition was a bit more complicated than I’d realized, though it doesn’t matter for what I’m saying in this […]