IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering

TL;DR

This paper introduces IPQA, a benchmark for core intent identification in personalized question answering, highlighting the challenge models face in understanding user priorities from behavior data.

Contribution

It proposes a novel benchmark and dataset for core intent identification in PQA, derived from user behavior and validated through systematic annotation.

Findings

Current models struggle with core intent identification in personalized contexts.

Model performance decreases as question complexity increases.

The dataset and code will be publicly available for future research.

Abstract

Intent identification serves as the foundation for generating appropriate responses in personalized question answering (PQA). However, existing benchmarks evaluate only response quality or retrieval performance without directly measuring intent identification capabilities. This gap is critical because without understanding which intents users prioritize, systems cannot generate responses satisfying individual information needs. To address this, we introduce the concept of core intents: intents users prioritize when selecting answers to satisfy their information needs. To evaluate these core intents, we propose IPQA, a benchmark for core Intent identification in Personalized Question Answering. Since users do not explicitly state their prioritized intents, we derive core intents from observable behavior patterns in answer selection, grounded in satisficing theory where users choose answers meeting their acceptance thresholds. We construct a dataset with various domains through systematic filtering, LLM-based annotation, and rigorous quality control combining automated verification with human validation. Experimental evaluations across state-of-the-art language models reveal that current systems struggle with core intent identification in personalized contexts. Models fail to identify core intents from user histories, with performance degrading as question complexity increases. The code and dataset will be made publicly available to facilitate future research in this direction.