SteerEval: A Framework for Evaluating Steerability with Natural Language Profiles for Recommendation

TL;DR

SteerEval is a new framework for evaluating how well natural-language user profiles can steer recommendations across diverse and nuanced user preferences, addressing limitations of existing benchmarks.

Contribution

We introduce SteerEval, a comprehensive evaluation framework for measuring nuanced steerability in natural-language recommender systems, highlighting its potential and limitations.

Findings

Natural-language profiles enable explicit user preference articulation.

SteerEval reveals varying effectiveness of steering across different content types.

Different intervention strategies impact steerability outcomes.

Abstract

Natural-language user profiles have recently attracted attention not only for improved interpretability, but also for their potential to make recommender systems more steerable. By enabling direct editing, natural-language profiles allow users to explicitly articulate preferences that may be difficult to infer from past behavior. However, it remains unclear whether current natural-language-based recommendation methods can follow such steering commands. While existing steerability evaluations have shown some success for well-recognized item attributes (e.g., movie genres), we argue that these benchmarks fail to capture the richer forms of user control that motivate steerable recommendations. To address this gap, we introduce SteerEval, an evaluation framework designed to measure more nuanced and diverse forms of steerability by using interventions that range from genres to content-warning for movies. We assess the steerability of a family of pretrained natural-language recommenders, examine the potential and limitations of steering on relatively niche topics, and compare how different profile and recommendation interventions impact steering effectiveness. Finally, we offer practical design suggestions informed by our findings and discuss future steps in steerable recommender design.