Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency

TL;DR

This paper critically examines the validity of existing summarization datasets, identifies factual inconsistencies, and introduces a new dataset, SummFC, with improved factual accuracy for better evaluation and training of summarization models.

Contribution

The paper highlights issues in current datasets, proposes a method to filter and improve factual consistency, and releases SummFC as a more reliable benchmark for summarization tasks.

Findings

Models trained on SummFC outperform those trained on unfiltered datasets.

SummFC dataset shows improved factual consistency in summaries.

The approach effectively identifies and filters problematic dataset instances.

Abstract

The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive summarization systems. However, the formulation of the task is rather ambiguous, neither the linguistic nor the natural language processing community has succeeded in giving a mutually agreed-upon definition. Due to this lack of well-defined formulation, a large number of popular abstractive summarization datasets are constructed in a manner that neither guarantees validity nor meets one of the most essential criteria of summarization: factual consistency. In this paper, we address this issue by combining state-of-the-art factual consistency models to identify the problematic instances present in popular summarization datasets. We release SummFC, a filtered summarization dataset with improved factual consistency, and demonstrate that models trained on this dataset achieve improved performance in nearly all quality aspects. We argue that our dataset should become a valid benchmark for developing and evaluating summarization systems.