Quantifying User Coherence: A Unified Framework for Analyzing Recommender Systems Across Domains

TL;DR

This paper introduces a unified, information-theoretic framework to analyze user behavior in recommender systems, revealing how user coherence impacts performance and guiding targeted system improvements.

Contribution

It proposes two novel measures, Mean Surprise and Mean Conditional Surprise, to quantify user profile characteristics and explain performance variability across algorithms and datasets.

Findings

Performance gains are concentrated on coherent users.

All algorithms perform poorly on incoherent users.

Measures enable targeted evaluation and system design.

Abstract

The performance of Recommender Systems (RS) varies significantly across users, yet the underlying reasons for this variance remain poorly understood. This paper introduces a unified framework to analyze and explain this performance gap by quantifying user profile characteristics. We propose two novel, information-theoretic measures: Mean Surprise (S(u)), which captures a user's deviation from popular items and is closely related to popularity bias, and Mean Conditional Surprise (CS(u)), which measures the internal coherence of a user's interactions in a domain-agnostic manner. Through extensive experiments on 7 algorithms and 9 datasets, we demonstrate that these measures are strong predictors of recommendation performance. Our analysis reveals that performance gains from complex models are concentrated on "coherent" users, while all algorithms perform poorly on "incoherent" users. We show how these measures provide practical utility for the Web community by: (1) enabling robust, stratified evaluation to identify model weaknesses; (2) facilitating a novel analysis of the behavioral alignment of recommendations; and (3) guiding targeted system design, which we validate by training a specialized model on a segment of "coherent" users that achieves superior performance for that group with significantly less data. This work provides a new lens for understanding user behavior and offers practical tools for building more robust and efficient large-scale recommender systems.