Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

TL;DR

This paper introduces PrefEval, a benchmark for assessing how well large language models can recognize, memorize, and follow user preferences in long conversations, revealing current limitations and potential improvements.

Contribution

The paper presents PrefEval, a comprehensive benchmark with 3,000 preference-query pairs, to evaluate and improve LLMs' ability to follow user preferences in multi-turn conversations.

Findings

State-of-the-art LLMs perform poorly in preference following, with accuracy below 10% at 10 turns.

Advanced prompting and retrieval methods only partially improve preference following.

Fine-tuning on PrefEval significantly enhances LLMs' ability to follow preferences.

Abstract

Large Language Models (LLMs) are increasingly used as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PrefEval, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PrefEval comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PrefEval contains user personalization or preference information in both explicit and implicit forms, and evaluates LLM performance using a generation and a classification task. With PrefEval, we evaluated the aforementioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in proactively following users' preferences during conversations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PrefEval significantly improves performance. We believe PrefEval serves as a valuable resource for measuring, understanding, and enhancing LLMs' preference following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https://prefeval.github.io/.