Self-Repetition in Abstractive Neural Summarizers

TL;DR

This paper investigates self-repetition in neural abstractive summarizers, analyzing how different architectures and training data influence repetitive outputs, and suggests ways to improve summarizer quality.

Contribution

It provides a quantitative analysis of self-repetition across architectures and datasets, highlighting factors that increase repetition and proposing methods to reduce it.

Findings

BART is more prone to self-repetition than T5 and Pegasus.

More abstractive and formulaic training data increases self-repetition.

Systems produce unrelated artefacts like ads and disclaimers.

Abstract

We provide a quantitative and qualitative analysis of self-repetition in the output of neural summarizers. We measure self-repetition as the number of n-grams of length four or longer that appear in multiple outputs of the same system. We analyze the behavior of three popular architectures (BART, T5, and Pegasus), fine-tuned on five datasets. In a regression analysis, we find that the three architectures have different propensities for repeating content across output summaries for inputs, with BART being particularly prone to self-repetition. Fine-tuning on more abstractive data, and on data featuring formulaic language, is associated with a higher rate of self-repetition. In qualitative analysis we find systems produce artefacts such as ads and disclaimers unrelated to the content being summarized, as well as formulaic phrases common in the fine-tuning domain. Our approach to corpus-level analysis of self-repetition may help practitioners clean up training data for summarizers and ultimately support methods for minimizing the amount of self-repetition.