To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text

TL;DR

This paper investigates how a popular neural summarization model balances copying and generating text, revealing its limited ability to produce truly abstractive and faithful summaries despite high ROUGE scores.

Contribution

The study provides a detailed analysis of the pointer-generator model's strategies and limitations in abstractive summarization, highlighting its reliance on syntactic cues and limited paraphrasing skills.

Findings

Model uses syntactic boundaries to truncate sentences on extractive-biased datasets.

Forcing generation reveals limited paraphrasing and factual inaccuracies.

Model shows limited abstractive abilities on abstractive-biased datasets.

Abstract

Abstractive neural summarization models have seen great improvements in recent years, as shown by ROUGE scores of the generated summaries. But despite these improved metrics, there is limited understanding of the strategies different models employ, and how those strategies relate their understanding of language. To understand this better, we run several experiments to characterize how one popular abstractive model, the pointer-generator model of See et al. (2017), uses its explicit copy/generation switch to control its level of abstraction (generation) vs extraction (copying). On an extractive-biased dataset, the model utilizes syntactic boundaries to truncate sentences that are otherwise often copied verbatim. When we modify the copy/generation switch and force the model to generate, only simple paraphrasing abilities are revealed alongside factual inaccuracies and hallucinations. On an abstractive-biased dataset, the model copies infrequently but shows similarly limited abstractive abilities. In line with previous research, these results suggest that abstractive summarization models lack the semantic understanding necessary to generate paraphrases that are both abstractive and faithful to the source document.