HorizonBench: Long-Horizon Personalization with Evolving Preferences

TL;DR

HorizonBench introduces a new benchmark and data generator for evaluating long-horizon personalization models that track evolving user preferences over six months.

Contribution

It provides a structured dataset and benchmark to diagnose and improve models' ability to handle long-term preference changes.

Findings

Best model achieves 52.8% accuracy, above chance.

Most models perform at or below 20% accuracy.

Models often fail to update preferences, defaulting to original stated values.

Abstract

User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench, a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ~4,300 turns and ~163K tokens. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user's originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization.