Revisiting Alternative Experimental Settings for Evaluating Top-N Item Recommendation Algorithms

TL;DR

This paper critically examines and compares different experimental setups for evaluating top-N item recommendation algorithms, providing guidelines to improve assessment accuracy.

Contribution

It systematically analyzes dataset splitting, sampled metrics, and domain selection, offering practical recommendations for more reliable evaluation of recommendation algorithms.

Findings

Different experimental settings significantly affect evaluation outcomes.

Recommendations for dataset splitting and metric sampling improve assessment reliability.

Guidelines help standardize evaluation practices in top-N recommendation research.

Abstract

Top-N item recommendation has been a widely studied task from implicit feedback. Although much progress has been made with neural methods, there is increasing concern on appropriate evaluation of recommendation algorithms. In this paper, we revisit alternative experimental settings for evaluating top-N recommendation algorithms, considering three important factors, namely dataset splitting, sampled metrics and domain selection. We select eight representative recommendation algorithms (covering both traditional and neural methods) and construct extensive experiments on a very large dataset. By carefully revisiting different options, we make several important findings on the three factors, which directly provide useful suggestions on how to appropriately set up the experiments for top-N item recommendation.