Learning Cluster Patterns for Abstractive Summarization

TL;DR

This paper introduces a clustering transformer layer that separates salient and non-salient context vectors in pre-trained models, leading to improved abstractive summarization performance.

Contribution

It proposes a novel clustering transformer layer that enhances the focus on salient information during summarization, outperforming existing models.

Findings

Up to 4% improvement in ROUGE scores

0.3% increase in BERTScore

Effective separation of salient and non-salient vectors

Abstract

Nowadays, pre-trained sequence-to-sequence models such as BERTSUM and BART have shown state-of-the-art results in abstractive summarization. In these models, during fine-tuning, the encoder transforms sentences to context vectors in the latent space and the decoder learns the summary generation task based on the context vectors. In our approach, we consider two clusters of salient and non-salient context vectors, using which the decoder can attend more to salient context vectors for summary generation. For this, we propose a novel clustering transformer layer between the encoder and the decoder, which first generates two clusters of salient and non-salient vectors, and then normalizes and shrinks the clusters to make them apart in the latent space. Our experimental result shows that the proposed model outperforms the existing BART model by learning these distinct cluster patterns, improving up to 4% in ROUGE and 0.3% in BERTScore on average in CNN/DailyMail and XSUM data sets.