CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

TL;DR

CoPA is a new benchmark for evaluating personalized question answering by measuring how well models align with individual user preferences derived from interaction data.

Contribution

It introduces a data-driven method to assess personalization in QA models using six cognitive factors and a benchmark with nearly 2,000 user profiles.

Findings

CoPA enables fine-grained, factor-level evaluation of personalized QA.

It provides a more comprehensive standard than generic metrics.

The benchmark correlates well with user-specific preferences.

Abstract

While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradigms predominantly rely on lexical-level similarity or manual heuristics, often lacking sufficient data-driven validation. We address this by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus, to distill six key personalization factors as evaluative dimensions. Accordingly, we introduce CoPA, a benchmark with 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA provides a more comprehensive and discriminative standard for evaluating personalized QA than generic metrics. The code is available at https://github.com/bjzgcai/CoPA.