Recommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets

TL;DR

This paper explores the problem of selecting the best recommender system algorithm for ranking prediction on implicit feedback datasets, using meta-models trained on extensive evaluations to improve prediction accuracy and algorithm recommendation.

Contribution

It introduces the first comprehensive evaluation of algorithm selection for ranking prediction on implicit feedback datasets and develops optimized meta-models that outperform existing automated methods.

Findings

Meta-models achieve high correlation (up to 0.918) with ground truth rankings.

Optimized meta-models significantly improve prediction of the best algorithm.

Traditional meta-models like XGBoost outperform automated ML models in recall for best algorithm prediction.

Abstract

The recommender systems algorithm selection problem for ranking prediction on implicit feedback datasets is under-explored. Traditional approaches in recommender systems algorithm selection focus predominantly on rating prediction on explicit feedback datasets, leaving a research gap for ranking prediction on implicit feedback datasets. Algorithm selection is a critical challenge for nearly every practitioner in recommender systems. In this work, we take the first steps toward addressing this research gap. We evaluate the NDCG@10 of 24 recommender systems algorithms, each with two hyperparameter configurations, on 72 recommender systems datasets. We train four optimized machine-learning meta-models and one automated machine-learning meta-model with three different settings on the resulting meta-dataset. Our results show that the predictions of all tested meta-models exhibit a median Spearman correlation ranging from 0.857 to 0.918 with the ground truth. We show that the median Spearman correlation between meta-model predictions and the ground truth increases by an average of 0.124 when the meta-model is optimized to predict the ranking of algorithms instead of their performance. Furthermore, in terms of predicting the best algorithm for an unknown dataset, we demonstrate that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.