Widespread Flaws in Offline Evaluation of Recommender Systems

TL;DR

This paper highlights common flaws in offline evaluation methods for recommender systems, emphasizing the need for more realistic and accurate evaluation setups to better predict online performance.

Contribution

It identifies four widespread evaluation flaws in recommender systems research and advocates for improved, more realistic offline evaluation practices.

Findings

Four common evaluation flaws identified

Flaws often copied without validation

Improving evaluation realism can enhance online performance prediction

Abstract

Even though offline evaluation is just an imperfect proxy of online performance -- due to the interactive nature of recommenders -- it will probably remain the primary way of evaluation in recommender systems research for the foreseeable future, since the proprietary nature of production recommenders prevents independent validation of A/B test setups and verification of online results. Therefore, it is imperative that offline evaluation setups are as realistic and as flawless as they can be. Unfortunately, evaluation flaws are quite common in recommender systems research nowadays, due to later works copying flawed evaluation setups from their predecessors without questioning their validity. In the hope of improving the quality of offline evaluation of recommender systems, we discuss four of these widespread flaws and why researchers should avoid them.