Medieval monarchies might not have had many things to recommend them compared with liberal democracies, but here's one: our rulers have no Fools. How often now will a national leader employ someone to laugh at their folly and remind them of bitter truths? More often, cabinets and advisers seem picked for their readiness to confirm their leader's judgements.

Some people fear that the information age encourages this tendency to spread to the rest of us. The Internet, they say, is a series of echo chambers: people join chat groups to hear others repeat their own opinions.

So climate sceptics talk only to other climate sceptics (and accuse climate scientists of doing likewise, perhaps with some justification). Elsewhere, the DailyMe.com website will supply you with only the news you ask to hear, realizing the vision of personalized news championed by Nicholas Negroponte of the Media Lab at the Massachusetts Institute of Technology in Cambridge. The 'Daily Me' is now often used in a pejorative sense to decry the insularity this inculcates.

Similarly, online shops tempt you with recommendations of 'more of the same', and music browsers such as Search Inside the Music aim to find you songs that 'sound similar' to ones you like already. But who's to say you wouldn't be more interested in stuff unlike what you like already?

That's the dilemma addressed in a paper in the Proceedings of the National Academy of Sciences by Yi-Cheng Zhang, a physicist at the University of Fribourg in Switzerland, and his co-workers1. They point out that most data-mining 'recommender' systems such as those used by Amazon.com focus on accuracy, measured by testing whether they can reproduce known user preferences. This emphasizes the similarity of recommendations to previous choices, and can lead to self-reinforcing cycles fixated on blockbuster items2.

But, say the researchers, the most useful recommendations may not be the most similar, but ones that offer the unexpected by introducing diversity. Like Lear's Fool, they challenge what you thought you knew.
The same but different

Zhang and colleagues show that a judicious blend of algorithms optimized for accuracy and for diversity can actually offer more of both than any of the component algorithms on their own. The researchers compare this effect with the value of 'weak ties' in our friendship networks. Although we tend to seek advice from close friends — typically people sharing similar views and preferences — it is often comments from people with whom we have a more limited connection that are the most helpful, because they offer a perspective outside our regular experience.

The same is true in scientific research: scientists from disciplines outside your own can spark new trains of thought, while your fellow specialists trudge along the same track. Without fertilization from outsiders, disciplines risk stultifying — which one recent study implies could be the fate of astronomy3.

Recommender systems that offer 'more of the same' can only encourage this sort of Balkanization of the ever-growing universe of information, opinion and choice.

If people truly want more of the same, it'll always be hard to make them hear the Fool's wisdom. But most recommender systems do want to find what people will like, not just what they think they like. Throwing diversity into the mix is a good start, but the bigger challenge is to figure out how preferences are formed. What are the coordinates of 'preference space' and how do we negotiate them? There might, say, be something about the melodic contours or timbres in Beethoven's music that a fan will find not in other early nineteenth-century composers but in twentieth-century modernists. Some music recommender systems are examining how we classify music according to non-traditional criteria, and using these as the compass directions for navigating music space (see 'Music to match your mood').

Understanding more about such preference-forming structures will not only improve the choices we're offered, but might also tell us something new about how the human brain partitions experience. And we could be in for some delicious surprises — just as when we used to browse in record stores.