Revisiting BPR: A Replicability Study of a Common Recommender System Baseline

TL;DR

This study revisits Bayesian Personalized Ranking (BPR), revealing implementation inconsistencies that affect performance and demonstrating that with proper tuning, BPR can match or surpass state-of-the-art recommender systems.

Contribution

The paper identifies discrepancies in open-source BPR implementations and shows that proper hyperparameter tuning significantly improves its performance.

Findings

Inconsistencies in open-source BPR implementations reduce performance by up to 50%.

Proper hyperparameter tuning enables BPR to match or outperform state-of-the-art methods.

BPR with tuning outperforms Mult-VAE by 10% in NDCG@100 on the Million Song Dataset.

Abstract

Bayesian Personalized Ranking (BPR), a collaborative filtering approach based on matrix factorization, frequently serves as a benchmark for recommender systems research. However, numerous studies often overlook the nuances of BPR implementation, claiming that it performs worse than newly proposed methods across various tasks. In this paper, we thoroughly examine the features of the BPR model, indicating their impact on its performance, and investigate open-source BPR implementations. Our analysis reveals inconsistencies between these implementations and the original BPR paper, leading to a significant decrease in performance of up to 50% for specific implementations. Furthermore, through extensive experiments on real-world datasets under modern evaluation settings, we demonstrate that with proper tuning of its hyperparameters, the BPR model can achieve performance levels close to state-of-the-art methods on the top-n recommendation tasks and even outperform them on specific datasets. Specifically, on the Million Song Dataset, the BPR model with hyperparameters tuning statistically significantly outperforms Mult-VAE by 10% in NDCG@100 with binary relevance function.