Pierre Henri Clouin's List: decision factory

Pandora, Last.fm, and iLike have broadened the appeal of recommendation algorithms and social discovery and sharing. Despite a challenging – even hostile - business environment, passionate entrepreneurs and music lovers are bringing to market new ways to explore and discover new music.

Mufin has gathered the most attention so far – being a spin off of the Fraunhofer Institut. Mufin relies exclusively on content analysis to make music recommendations. The service analyzes 40 characteristics for each the 4 million songs in its database, including tempo, sound density, and variety of other factors to filter out songs which are similar. This content-based approach uniquely addresses cold start and long tail discovery issues, but feels a bit too “literal” in the way it analyzes music similarities between songs.

The perceptron is an experimental and transparent collaborative filtering service. The service aggregates and filters recommendations made by actual humans (from tinymixtapes.com and epitonic.com), social links between artists and fans (e.g. belonging to the same label or the number of friends on a myspace page), and other social music sharing sources. Interestingly, recommendations sources and the relative weights of the various sources are all public. The service does a great job at mixing familiar and unknown recommendations, thereby making the new stuff all the more interesting. It would be terrific to see a next generation of this service allow users to actually tinker with sources and weights to obtain more personalized recommendations.

The next big sound pushes the music discovery envelope. The service does a great job at crowd-sourcing and uncovering new musical talents, using the music label paradigm. Each listener is allowed to select and sign up to 10 artists at any given time among all the unsigned artists that broadcast their music on the site. This site is a great place for unadulterated creative talent, although a bit of collaborative filtering would enhance the listening experience.

With Bla

Gabe Rivera of Techmeme fame has sparked a lively debate as he announced he had hired an editor to improve interestingness, reactivity and relevance of his news aggregation website (see VentureBeat, TechCrunch, and ReadWriteWeb). Rivera’s plight echoes Netflix’s challenges at boosting its movie recommendation engine, as algorithmic improvements gradually near their natural asymptote.

Techmeme is good at aggregating news overtime but not at breaking them, pointing to several common datamining and recommendation challenges:

- “Cold start” – interestingly digg’s social voting approach hasn’t been able to overcome that challenge either – in both cases, the services cannot anticipate news’ propagation and velocity;

- Context – Techmeme sometimes mixes up headlines and for instance ended up featuring news about Anna Nicole Smith’s hospitalization after she’s already been declared dead;

- Outliers or the “Napoleon Dynamite” problem, as the New York Times dubs it - identifying newsworthy pieces from uncommon sources before they make it into the mainstream is also an issue.

Interestingness and relevance are Techmeme’s other key reasons to bring in a human eye. Techmeme bets that an expert hand can be a better judge than crowdsourced implicit feedback based on clickstream or explicit feedback such as social voting. This approach seems to contradict much of the crowdsourcing mantra, although Techmeme’s case is more about rebalancing than shunning crowdsourcing.

For online retailers, Techmeme’s move to “curated news aggregation” highlights opportunities to blend human input, datamining, and recommendation:

- to add context to a product recommendation – based on usage, audience background, as well as internal needs through promotions;

- to identify new and unexplored relationships between products – for product discovery, up-sell, and outliers.

For the second year, Netflix has awarded a $50K progress prize. The winning team – BellKor in BigChaos – improved on Netflix’s recommendation algorithm by 9.44%. Although the 10% improvement seems to be getting closer in absolute terms, the last stretch might prove elusive for a little while longer.

Is Netflix’s ground-breaking crowdsourcing approach reaching a limit? Netflix has undeniably got a lot of value out of their contest – 10,000s people working and an algorithmic improvement very close to 10% – for only $100K so far. However, the winning method this year – and the likely research directions for next year – have exponential data and computing power requirements – possibly putting the contest out of reach of most casual researchers.

Are “curated” recommendations superior to pure statistics? Curated recommendations have received some hype lately – from Techmeme’s hiring of an editor to the coincidental launch this week of ClerkDogs – a “clerks-in-a-box” online movie recommendation service? A human touch can certainly rebalance or add a forward-looking perspective to recommendations where data sets are ambiguous (such as clickstream), too small, or lack enough historical data.

Apple has quietly released a few changes to its iPhone App Store on iTunes, in an attempt to alleviate some of the growing pains around its app ecosystem. The App Store has recently hit 10,000 apps and is expected to accept many more.

“Most popular” lists and 19 high-level categories are hardly up to the task of helping consumers find new apps. Limited discovery is hampering the App Store’s growth and monetization and could have a depressing effect on the ecosystem.

There are however quite a few opportunities for Apple to improve discovery and bring the overall shopping experience on the App Store away from simply being a “port” of the iTunes Musicstore:

- Improve discovery through experience

- Enable free trials – free trials are critical to bring feature-rich and higher priced apps to the ecosystem.

- Add videos and user video reviews – text and screenshots do little justice to an app’s capabilities, design, and overall user experience;

- Improve discovery through richer feedback and needs analysis

- Categories and search are limited tools for demand generation;

- Amazon-style recommendations and discovery needs to be more ubiquitous and prominent on the App Store to enable genuine comparison between apps;

- Blending the feedback process about apps much with the user experience on the iPhone could open up a much more personalized experience

- Improve discovery through finer segmentation and filtering – the free vs. paid segmentation is dragging all paid iPhone apps down in the ringtones pricing range; there are only 14 apps over $100 (out of 10,000) and there are still relatively few games at a price comparable to that of a typical game console.