The problem with recommendations

So I have taken to listening to a number of NPR shows via podcast while on the subway to and from work. One of my more recent additions is On the Media, a weekly media analysis program. This week's program had a number of interesting segments, but the one that most caught my attention by far was this piece on Netflix's recommendation engine. It's an interview with technology writer Clive Thompson, who's discussing his article for the New York Times Magazine on the subject, which I've only just started reading. Very interesting listen; there's also a transcript.

Two years ago, Netflix offered a million dollars to anyone who could improve their recommendation engine by 10%. No one has won yet; the closest team is within 9.5%, but progress has been slow -- turns out it was easy to gain a 5% improvement, but every step closer to 10% has been harder and harder. And the trouble seems to center around a very certain type of film -- the quirky indy comedy. Thompson quotes one of the programmers competing in the contest, who claims that his particular stumbling block is the movie Napoleon Dynamite. From the NYT Magazine article:

The reason, Bertoni says, is that "Napoleon Dynamite" is very weird and very polarizing.... It’s the type of quirky entertainment that tends to be either loved or despised. The movie has been rated more than two million times in the Netflix database, and the ratings are disproportionately one or five stars. Worse, close friends who normally share similar film aesthetics often heatedly disagree about whether "Napoleon Dynamite" is a masterpiece or an annoying bit of hipster self-indulgence. When Bertoni saw the movie himself with a group of friends, they argued for hours over it. "Half of them loved it, and half of them hated it," he told me. "And they couldn’t really say why. It’s just a difficult movie."

...Bertoni has deduced that ["Napoleon Dynamite"] is causing 15 percent of his remaining error rate.... And while "Napoleon Dynamite" is the worst culprit, it isn’t the only troublemaker. A small subset of other titles have caused almost as much bedevilment among the Netflix Prize competitors. When Bertoni showed me a list of his 25 most-difficult-to-predict movies, I noticed they were all similar in some way to "Napoleon Dynamite" — culturally or politically polarizing and hard to classify, including "I Heart Huckabees," "Lost in Translation," "Fahrenheit 9/11," "The Life Aquatic With Steve Zissou," "Kill Bill: Volume 1" and "Sideways."

The question of recommendation engines, and how they work and why they sometimes fail so badly, is an interesting one to me -- professionally as well as personally, since librarians are often asked for media recommendations. So far, the best tool of that type I know is Pandora (the music recommendation engine based on the Music Genome Project), but I'm not sure how you would extend that to books or movies. Thompson never mentioned Pandora in his interview, and I haven't gotten far enough yet in the NYT Magazine article to know if it's been mentioned, but I'd be interested to see if it could be applied to the Netflix problem.