Entity-level Factual Consistency of Abstractive Text Summarization

TL;DR

This paper introduces new metrics for evaluating entity-level factual consistency in abstractive summarization, identifies data filtering as a solution to entity hallucination, and proposes joint entity-summary generation methods for improved accuracy.

Contribution

It presents novel metrics for entity-level factuality, demonstrates data filtering reduces hallucination, and introduces joint entity and summary generation techniques.

Findings

New metrics effectively measure entity-level factual consistency.

Filtering training data reduces entity hallucination.

Joint entity-summary generation improves factual accuracy.

Abstract

A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.