Do recommender systems benefit users?

TL;DR

This paper models the benefit of recommender systems for users, revealing that over-reliance can negate benefits, and highlights the discrepancy between common accuracy metrics and actual user benefit.

Contribution

Introduces a model to evaluate user benefits from recommender systems, demonstrating the importance of balanced reliance and accuracy metrics.

Findings

Recommendations can be equivalent to random draws if over-relied upon.

High accuracy metrics do not always mean high user benefit.

An improved recommendation approach with better accuracy is proposed.

Abstract

Recommender systems are present in many web applications to guide our choices. They increase sales and benefit sellers, but whether they benefit customers by providing relevant products is questionable. Here we introduce a model to examine the benefit of recommender systems for users, and found that recommendations from the system can be equivalent to random draws if one relies too strongly on the system. Nevertheless, with sufficient information about user preferences, recommendations become accurate and an abrupt transition to this accurate regime is observed for some algorithms. On the other hand, we found that a high accuracy evaluated by common accuracy metrics does not necessarily correspond to a high real accuracy nor a benefit for users, which serves as an alarm for operators and researchers of recommender systems. We tested our model with a real dataset and observed similar behaviors. Finally, a recommendation approach with improved accuracy is suggested. These results imply that recommender systems can benefit users, but relying too strongly on the system may render the system ineffective.