User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction

TL;DR

This paper introduces VARS, a framework that personalizes conversational LLM agents by representing user preferences with vectors updated through weak feedback, improving interaction efficiency and interpretability without fine-tuning.

Contribution

VARS is a novel, pipeline-agnostic approach that models user preferences with long-term and short-term vectors, enabling effective personalization through online updates from weak scalar rewards.

Findings

VARS improves interaction efficiency over raw task accuracy.

The long-term vectors align with cross-user preferences.

Short-term vectors adapt to session-specific needs.

Abstract

Large language models are increasingly used as personal assistants, yet most lack a persistent user model, forcing users to repeatedly restate preferences across sessions. We propose Vector-Adapted Retrieval Scoring (VARS), a pipeline-agnostic, frozen-backbone framework that represents each user with long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory. The vectors are updated online from weak scalar rewards from users' feedback, enabling personalization without per-user fine-tuning. We evaluate on \textsc{MultiSessionCollab}, an online multi-session collaboration benchmark with rich user preference profiles, across math and code tasks. Under frozen backbones, the main benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy: our full VARS agent achieves the strongest overall performance, matches a strong Reflection baseline in task success, and reduces timeout rate and user effort. The learned long-term vectors also align with cross-user preference overlap, while short-term vectors capture session-specific adaptation, supporting the interpretability of the dual-vector design. Code, model, and data are available at https://github.com/YurenHao0426/VARS.