The Hidden Cost of Defaults in Recommender System Evaluation

TL;DR

This paper reveals that default settings in the RecBole recommendation framework, especially undocumented early-stopping policies, can bias hyperparameter optimization, affecting reproducibility and performance evaluation.

Contribution

It uncovers hidden default behaviors in RecBole that influence hyperparameter search outcomes and emphasizes the need for transparent, reproducibility-aware recommendation frameworks.

Findings

Default early-stopping can prematurely terminate searches.

Hidden framework logic causes variability comparable to search strategy differences.

Recommendations for improving transparency and reproducibility in recommender system evaluation.

Abstract

Hyperparameter optimization is critical for improving the performance of recommender systems, yet its implementation is often treated as a neutral or secondary concern. In this work, we shift focus from model benchmarking to auditing the behavior of RecBole, a widely used recommendation framework. We show that RecBole's internal defaults, particularly an undocumented early-stopping policy, can prematurely terminate Random Search and Bayesian Optimization. This limits search coverage in ways that are not visible to users. Using six models and two datasets, we compare search strategies and quantify both performance variance and search path instability. Our findings reveal that hidden framework logic can introduce variability comparable to the differences between search strategies. These results highlight the importance of treating frameworks as active components of experimental design and call for more transparent, reproducibility-aware tooling in recommender systems research. We provide actionable recommendations for researchers and developers to mitigate hidden configuration behaviors and improve the transparency of hyperparameter tuning workflows.