Improving Abstraction in Text Summarization

TL;DR

This paper introduces methods to enhance abstraction in text summarization by decomposing the decoder and optimizing for novel phrase generation, resulting in summaries that are more abstract yet comparable in quality to existing models.

Contribution

It presents a novel decoder architecture and a new metric for abstraction, improving the level of novelty in generated summaries.

Findings

Achieves higher abstraction levels compared to existing models.

Maintains competitive ROUGE scores and human evaluation results.

Successfully encourages the generation of novel phrases.

Abstract

Abstractive text summarization aims to shorten long text documents into a human readable form that contains the most important facts from the original document. However, the level of actual abstraction as measured by novel phrases that do not appear in the source document remains low in existing approaches. We propose two techniques to improve the level of abstraction of generated summaries. First, we decompose the decoder into a contextual network that retrieves relevant parts of the source document, and a pretrained language model that incorporates prior knowledge about language generation. Second, we propose a novelty metric that is optimized directly through policy learning to encourage the generation of novel phrases. Our model achieves results comparable to state-of-the-art models, as determined by ROUGE scores and human evaluations, while achieving a significantly higher level of abstraction as measured by n-gram overlap with the source document.