Are Recommenders Self-Aware? Label-Free Recommendation Performance Estimation via Model Uncertainty

TL;DR

This paper introduces LiDu, a probability-based method to estimate recommendation model performance without labels by measuring uncertainty, enhancing transparency and self-awareness in recommender systems.

Contribution

It proposes LiDu, a novel uncertainty measure for recommenders, validated on synthetic and real datasets, linking uncertainty with performance.

Findings

LiDu correlates better with performance than existing estimators.

LiDu provides insights into model states during training and inference.

The method enhances transparency and self-awareness of recommenders.

Abstract

Can a recommendation model be self-aware? This paper investigates the recommender's self-awareness by quantifying its uncertainty, which provides a label-free estimation of its performance. Such self-assessment can enable more informed understanding and decision-making before the recommender engages with any users. To this end, we propose an intuitive and effective method, probability-based List Distribution uncertainty (LiDu). LiDu measures uncertainty by determining the probability that a recommender will generate a certain ranking list based on the prediction distributions of individual items. We validate LiDu's ability to represent model self-awareness in two settings: (1) with a matrix factorization model on a synthetic dataset, and (2) with popular recommendation algorithms on real-world datasets. Experimental results show that LiDu is more correlated with recommendation performance than a series of label-free performance estimators. Additionally, LiDu provides valuable insights into the dynamic inner states of models throughout training and inference. This work establishes an empirical connection between recommendation uncertainty and performance, framing it as a step towards more transparent and self-evaluating recommender systems.