Automatic Text Extractive Summarization Based on Graph and Pre-trained Language Model Attention

TL;DR

This paper introduces a novel extractive summarization method that integrates graph representations of text with attention mechanisms from pre-trained language models, utilizing GCNs for sentence salience detection.

Contribution

It proposes combining attention matrices from pre-trained models with graph convolutional networks to improve extractive summarization performance.

Findings

Achieves competitive results on benchmark datasets.

Effectively identifies salient sentences using the proposed graph-attention approach.

Demonstrates the integration of attention matrices and GCNs enhances summarization quality.

Abstract

Representing a text as a graph for obtaining automatic text summarization has been investigated for over ten years. With the development of attention or Transformer on natural language processing (NLP), it is possible to make a connection between the graph and attention structure for a text. In this paper, an attention matrix between the sentences of the whole text is adopted as a weighted adjacent matrix of a fully connected graph of the text, which can be produced through the pre-training language model. The GCN is further applied to the text graph model for classifying each node and finding out the salient sentences from the text. It is demonstrated by the experimental results on two typical datasets that our proposed model can achieve a competitive result in comparison with sate-of-the-art models.