Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

TL;DR

This paper systematically compares ten collaborative filtering algorithms, including neural and traditional models, across multiple datasets and metrics, revealing no clear overall best and emphasizing the importance of proper baselines in top-N recommendation research.

Contribution

It provides a comprehensive, reproducible benchmark of various algorithms, highlighting the inconsistent performance of neural models and offering fine-tuned baselines for future research.

Findings

No single algorithm dominates across datasets and metrics.

Neural models do not outperform traditional methods in accuracy.

Linear, nearest-neighbor, and matrix factorization methods perform reliably.

Abstract

Research on recommender systems algorithms, like other areas of applied machine learning, is largely dominated by efforts to improve the state-of-the-art, typically in terms of accuracy measures. Several recent research works however indicate that the reported improvements over the years sometimes "don't add up", and that methods that were published several years ago often outperform the latest models when evaluated independently. Different factors contribute to this phenomenon, including that some researchers probably often only fine-tune their own models but not the baselines. In this paper, we report the outcomes of an in-depth, systematic, and reproducible comparison of ten collaborative filtering algorithms - covering both traditional and neural models - on several common performance measures on three datasets which are frequently used for evaluation in the recent literature. Our results show that there is no consistent winner across datasets and metrics for the examined top-n recommendation task. Moreover, we find that for none of the accuracy measurements any of the considered neural models led to the best performance. Regarding the performance ranking of algorithms across the measurements, we found that linear models, nearest-neighbor methods, and traditional matrix factorization consistently perform well for the evaluated modest-sized, but commonly-used datasets. Our work shall therefore serve as a guideline for researchers regarding existing baselines to consider in future performance comparisons. Moreover, by providing a set of fine-tuned baseline models for different datasets, we hope that our work helps to establish a common understanding of the state-of-the-art for top-n recommendation tasks.