CiteEval: Principle-Driven Citation Evaluation for Source Attribution

TL;DR

CiteEval introduces a principle-driven framework for more nuanced citation evaluation, incorporating context and user intent, supported by a new benchmark and model-based metrics that outperform existing methods.

Contribution

The paper presents a novel, principle-based citation evaluation framework, a comprehensive benchmark dataset, and model metrics that better align with human judgments than prior approaches.

Findings

CiteEval-Auto correlates strongly with human judgments.

The framework captures multifaceted citation quality aspects.

Model metrics outperform existing automatic evaluation methods.

Abstract

Citation quality is crucial in information-seeking systems, directly influencing trust and the effectiveness of information access. Current evaluation frameworks, both human and automatic, mainly rely on Natural Language Inference (NLI) to assess binary or ternary supportiveness from cited sources, which we argue is a suboptimal proxy for citation evaluation. In this work we introduce CiteEval, a citation evaluation framework driven by principles focusing on fine-grained citation assessment within a broad context, encompassing not only the cited sources but the full retrieval context, user query, and generated text. Guided by the proposed framework, we construct CiteBench, a multi-domain benchmark with high-quality human annotations on citation quality. To enable efficient evaluation, we further develop CiteEval-Auto, a suite of model-based metrics that exhibit strong correlation with human judgments. Experiments across diverse systems demonstrate CiteEval-Auto's superior ability to capture the multifaceted nature of citations compared to existing metrics, offering a principled and scalable approach to evaluate and improve model-generated citations.