Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation

TL;DR

This paper critically evaluates automatic factuality metrics for summarization, revealing their limitations in detecting deep factual inconsistencies and their vulnerability to manipulation, thus questioning their reliability.

Contribution

The study systematically tests various factuality metrics, including LLM-based prompts, exposing their weaknesses and potential for gaming, and highlights the need for more robust evaluation methods.

Findings

Metrics perform poorly on complex, reasoning-required cases.

Some metrics are sensitive to benign edits, not factual errors.

Prompt-based LLM assessments can be manipulated by content-free additions.

Abstract

Modern LLMs can now produce highly readable abstractive summaries, to the point that traditional automated metrics for evaluating summary quality, such as ROUGE, have saturated. However, LLMs still sometimes introduce inaccuracies into summaries, i.e., information inconsistent with or unsupported by the corresponding source. Measuring the occurrence of these often subtle factual inconsistencies automatically has proved challenging. This in turn has motivated development of metrics intended to measure the factual consistency of generated summaries against sources. But are these approaches measuring what they purport to? Or are they mostly exploiting artifacts? In this work, we stress test a range of automatic factuality metrics, including specialized models and LLM-based prompting methods, to probe what they actually capture. Using a shallow classifier to separate ``easy'' examples for factual evaluation where surface features suffice from ``hard'' cases requiring deeper reasoning, we find that all metrics show substantial performance drops on the latter. Furthermore, some metrics are more sensitive to benign, fact-preserving edits than to factual corrections. Building on this observation, we demonstrate that most automatic factuality metrics can be gamed, i.e., their scores can be artificially inflated by appending innocuous, content-free sentences to summaries. Among the metrics tested, the prompt based ChatGPT-DA approach is the most robust and reliable. However, this comes with a notable caveat: Prompting LLMs to assess factuality may overly rely on their parametric knowledge rather than the provided reference when making judgments. Taken together, our findings call into question the reliability of current factuality metrics and prompt a broader reflection on what these metrics are truly measuring.