BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs

TL;DR

BenchPreS is a benchmark designed to evaluate how well large language models apply or suppress stored user preferences appropriately across different communication contexts, highlighting current limitations in context sensitivity.

Contribution

The paper introduces BenchPreS, a novel benchmark with metrics to assess context-aware preference application in persistent-memory LLMs, revealing their struggles with context sensitivity.

Findings

Frontier LLMs often misapply preferences in contextually inappropriate situations.

Stronger preference adherence correlates with higher over-application rates.

Reasoning skills and prompt defenses do not fully mitigate preference misapplication.

Abstract

Large language models (LLMs) increasingly store user preferences in persistent memory to support personalization across interactions. However, in third-party communication settings governed by social and institutional norms, some user preferences may be inappropriate to apply. We introduce BenchPreS, which evaluates whether memory-based user preferences are appropriately applied or suppressed across communication contexts. Using two complementary metrics, Misapplication Rate (MR) and Appropriate Application Rate (AAR), we find even frontier LLMs struggle to apply preferences in a context-sensitive manner. Models with stronger preference adherence exhibit higher rates of over-application, and neither reasoning capability nor prompt-based defenses fully resolve this issue. These results suggest current LLMs treat personalized preferences as globally enforceable rules rather than as context-dependent normative signals.