FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

TL;DR

FENICE is a new, interpretable, and efficient factuality metric for summarization that uses natural language inference and claim extraction to improve factual consistency evaluation, outperforming existing benchmarks especially on long summaries.

Contribution

The paper introduces FENICE, a novel factuality evaluation metric combining NLI and claim extraction, addressing interpretability and efficiency issues of prior methods.

Findings

FENICE achieves state-of-the-art results on AGGREFACT benchmark.

It effectively evaluates factuality in long-form summaries.

The method is more interpretable and computationally practical.

Abstract

Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization. In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at https://github.com/Babelscape/FENICE.