On the Accuracy Limits of Sequential Recommender Systems: An Entropy-Based Approach

TL;DR

This paper introduces an entropy-based, model-agnostic method to estimate the intrinsic accuracy limit of sequential recommender systems, aiding in difficulty assessment and data optimization.

Contribution

It develops a training-free estimator that accurately predicts recommendation difficulty, remains insensitive to candidate set size, and supports user-group diagnostics.

Findings

Estimator tracks oracle-controlled difficulty more faithfully than baselines.

Remains insensitive to candidate-set size.

Supports user-group diagnostics revealing predictability differences.

Abstract

Sequential recommender systems have achieved steady gains in offline accuracy, yet it remains unclear how close current models are to the intrinsic accuracy limit imposed by the data. A reliable, model-agnostic estimate of this ceiling would enable principled difficulty assessment and headroom estimation before costly model development. Existing predictability analyses typically combine entropy estimation with Fano's inequality inversion; however, in recommendation they are hindered by sensitivity to candidate-space specification and distortion from Fano-based scaling in low-predictability regimes. We develop an entropy-induced, training-free approach for quantifying accuracy limits in sequential recommendation, yielding a candidate-size-agnostic estimate. Experiments on controlled synthetic generators and diverse real-world benchmarks show that the estimator tracks oracle-controlled difficulty more faithfully than baselines, remains insensitive to candidate-set size, and achieves high rank consistency with best-achieved offline accuracy across state-of-the-art sequential recommenders (Spearman rho up to 0.914). It also supports user-group diagnostics by stratifying users by novelty preference, long-tail exposure, and activity, revealing systematic predictability differences. Furthermore, predictability can guide training data selection: training sets constructed from high-predictability users yield strong downstream performance under reduced data budgets. Overall, the proposed estimator provides a practical reference for assessing attainable accuracy limits, supporting user-group diagnostics, and informing data-centric decisions in sequential recommendation.