Towards Fair and Rigorous Evaluations: Hyperparameter Optimization for Top-N Recommendation Task with Implicit Feedback

TL;DR

This paper investigates hyperparameter optimization for Top-N implicit recommendation models, emphasizing fair evaluation practices and identifying effective search algorithms across datasets to improve reproducibility and comparison.

Contribution

It introduces a systematic methodology for hyperparameter tuning in recommender systems, promoting fair comparisons and reproducibility in research.

Findings

Identified the most suitable hyperparameter search algorithms for different recommendation models.

Provided a benchmark for hyperparameter optimization in Top-N implicit recommendation tasks.

Enhanced reproducibility by standardizing hyperparameter settings in experiments.

Abstract

The widespread use of the internet has led to an overwhelming amount of data, which has resulted in the problem of information overload. Recommender systems have emerged as a solution to this problem by providing personalized recommendations to users based on their preferences and historical data. However, as recommendation models become increasingly complex, finding the best hyperparameter combination for different models has become a challenge. The high-dimensional hyperparameter search space poses numerous challenges for researchers, and failure to disclose hyperparameter settings may impede the reproducibility of research results. In this paper, we investigate the Top-N implicit recommendation problem and focus on optimizing the benchmark recommendation algorithm commonly used in comparative experiments using hyperparameter optimization algorithms. We propose a research methodology that follows the principles of a fair comparison, employing seven types of hyperparameter search algorithms to fine-tune six common recommendation algorithms on three datasets. We have identified the most suitable hyperparameter search algorithms for various recommendation algorithms on different types of datasets as a reference for later study. This study contributes to algorithmic research in recommender systems based on hyperparameter optimization, providing a fair basis for comparison.