The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection

TL;DR

This paper introduces a new framework and benchmark for detecting unfaithfulness in LLM-generated summaries, addressing annotation ambiguity by including an intermediate category and revealing significant hallucination issues.

Contribution

We propose an annotation framework with an Out-Dependent category and create VeriGray, a challenging benchmark for unfaithfulness detection in summarization tasks.

Findings

SOTA LLMs hallucinate about 6% of sentences in summaries.

Approximately 9% of generated sentences require external knowledge for verification.

Our benchmark challenges existing detection methods, highlighting room for improvement.

Abstract

Ensuring that Large Language Models (LLMs) generate summaries faithful to a given source document is essential for real-world applications. While prior research has explored LLM faithfulness, existing benchmarks suffer from annotation ambiguity, primarily due to the ill-defined boundary of permissible external knowledge in generated outputs. For instance, common sense is often incorporated into responses and labeled as "faithful", yet the acceptable extent of such knowledge remains unspecified, leading to inconsistent annotations. To address this issue, we propose a novel faithfulness annotation framework, which introduces an intermediate category, Out-Dependent, to classify cases where external knowledge is required for verification. Using this framework, we construct VeriGray (Verification with the Gray Zone) -- a new unfaithfulness detection benchmark in summarization. Statistics reveal that even SOTA LLMs, such as GPT-5, exhibit hallucinations ($\sim 6\%$ of sentences) in summarization tasks. Moreover, a substantial proportion ($\sim 9\%$ on average of models) of generated sentences fall into the Out-Dependent category, underscoring the importance of resolving annotation ambiguity in unfaithfulness detection benchmarks. Experiments demonstrate that our benchmark poses significant challenges to multiple baseline methods, indicating considerable room for future improvement.