Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models

TL;DR

This paper explores combining saliency models with pre-trained sequence-to-sequence models to improve abstractive summarization, demonstrating that such integration enhances performance over standard fine-tuning methods.

Contribution

It introduces a novel combination model that integrates saliency detection with seq-to-seq models, achieving superior results on benchmark datasets.

Findings

Combination models outperform simple fine-tuned seq-to-seq models.

The proposed model exceeds previous best by 1.33 ROUGE-L points on CNN/DM.

Most models outperform baseline on both datasets.

Abstract

Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although the fluency of abstractive summarization has been greatly improved by fine-tuning these models, it is not clear whether they can also identify the important parts of the source text to be included in the summary. In this study, we investigated the effectiveness of combining saliency models that identify the important parts of the source text with the pre-trained seq-to-seq models through extensive experiments. We also proposed a new combination model consisting of a saliency model that extracts a token sequence from a source text and a seq-to-seq model that takes the sequence as an additional input text. Experimental results showed that most of the combination models outperformed a simple fine-tuned seq-to-seq model on both the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination model exceeded the previous best-performed model by 1.33 points on ROUGE-L.