Citation Failure: Definition, Analysis and Efficient Mitigation

TL;DR

This paper analyzes citation failure in LLM-based RAG systems, introduces benchmarks to study it, and proposes a framework to mitigate it effectively, improving citation quality.

Contribution

It introduces CITECONTROL benchmark for failure analysis and CITENTION framework for efficient citation mitigation in LLMs.

Findings

Citation failures increase with relational complexity.

Combining citation methods improves performance.

Proposed framework significantly enhances citation quality.

Abstract

Citations from LLM-based RAG systems are supposed to simplify response verification. However, this goal is undermined in cases of citation failure, where a model generates a helpful response, but fails to generate citations to complete evidence. In contrast to previous work, we propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible. To address citation failure, this work follows a two-step approach: (1) We study when citation failure occurs and (2) how it can be mitigated efficiently. For step 1, we extend prior work by investigating how the relation between response and evidence affects citation quality. We introduce CITECONTROL, a benchmark that systematically varies this relation to enable the analysis of failure modes. Experiments show that failures increase with relational complexity and suggest that combining citation methods could improve performance, motivating step 2. To study the efficient improvement of LLM citation, we propose CITENTION, a framework integrating generative, attention-based, and retrieval-based methods. Results demonstrate substantial citation improvements on CITECONTROL and in transfer settings. We make our data and code publicly available.