Counterfactual Data Augmentation improves Factuality of Abstractive Summarization

TL;DR

This paper introduces counterfactual data augmentation techniques that enhance the factual accuracy of abstractive summarization models without compromising their overall quality, demonstrated on popular datasets.

Contribution

The authors propose three novel augmentation methods using entity replacement and hypernym substitution to improve factual correctness in summarization.

Findings

Factual correctness improved by about 2.5 points on CNN/Dailymail and XSum datasets.

Augmentation does not significantly affect ROUGE scores.

Methods increase training data diversity and factual consistency.

Abstract

Abstractive summarization systems based on pretrained language models often generate coherent but factually inconsistent sentences. In this paper, we present a counterfactual data augmentation approach where we augment data with perturbed summaries that increase the training data diversity. Specifically, we present three augmentation approaches based on replacing (i) entities from other and the same category and (ii) nouns with their corresponding WordNet hypernyms. We show that augmenting the training data with our approach improves the factual correctness of summaries without significantly affecting the ROUGE score. We show that in two commonly used summarization datasets (CNN/Dailymail and XSum), we improve the factual correctness by about 2.5 points on average