Rank-Preference Consistency as the Appropriate Metric for Recommender Systems

TL;DR

This paper argues that traditional rating prediction accuracy metrics are inadequate for evaluating recommender systems and proposes rank-preference consistency as a more appropriate measure for assessing how well RS models predict user preferences.

Contribution

The paper introduces rank-preference consistency as a novel evaluation metric that directly measures the accuracy of user preference predictions in recommender systems.

Findings

Traditional metrics like RMSE do not effectively predict user preferences.

Methods optimized for RMSE are often ineffective at capturing true user preferences.

Rank-preference consistency provides a more meaningful assessment of RS performance.

Abstract

In this paper we argue that conventional unitary-invariant measures of recommender system (RS) performance based on measuring differences between predicted ratings and actual user ratings fail to assess fundamental RS properties. More specifically, posing the optimization problem as one of predicting exact user ratings provides only an indirect suboptimal approximation for what RS applications typically need, which is an ability to accurately predict user preferences. We argue that scalar measures such as RMSE and MAE with respect to differences between actual and predicted ratings are only proxies for measuring RS ability to accurately estimate user preferences. We propose what we consider to be a measure that is more fundamentally appropriate for assessing RS performance, rank-preference consistency, which simply counts the number of prediction pairs that are inconsistent with the user's expressed product preferences. For example, if an RS predicts the user will prefer product A over product B, but the user's withheld ratings indicate s/he prefers product B over A, then rank-preference consistency has been violated. Our test results conclusively demonstrate that methods tailored to optimize arbitrary measures such as RMSE are not generally effective at accurately predicting user preferences. Thus, we conclude that conventional methods used for assessing RS performance are arbitrary and misleading.