ALPBench: A Benchmark for Attribution-level Long-term Personal Behavior Understanding

TL;DR

ALPBench is a new benchmark designed to evaluate large language models' ability to understand and predict long-term user preferences at an attribution level, focusing on complex attribute interactions and reasoning over user history.

Contribution

It introduces a novel attribution-level benchmark that assesses LLMs' capacity for long-term personal behavior understanding through attribute combination prediction.

Findings

ALPBench enables detailed evaluation of personalization capabilities.

Current LLMs struggle with complex attribute interaction reasoning.

The benchmark emphasizes long-term behavior modeling over explicit user requests.

Abstract

Recent advances in large language models have highlighted their potential for personalized recommendation, where accurately capturing user preferences remains a key challenge. Leveraging their strong reasoning and generalization capabilities, LLMs offer new opportunities for modeling long-term user behavior. To systematically evaluate this, we introduce ALPBench, a Benchmark for Attribution-level Long-term Personal Behavior Understanding. Unlike item-focused benchmarks, ALPBench predicts user-interested attribute combinations, enabling ground-truth evaluation even for newly introduced items. It models preferences from long-term historical behaviors rather than users' explicitly expressed requests, better reflecting enduring interests. User histories are represented as natural language sequences, allowing interpretable, reasoning-based personalization. ALPBench enables fine-grained evaluation of personalization by focusing on the prediction of attribute combinations task that remains highly challenging for current LLMs due to the need to capture complex interactions among multiple attributes and reason over long-term user behavior sequences.