Attention Optimization for Abstractive Document Summarization

TL;DR

This paper introduces an enhanced attention mechanism for abstractive document summarization, utilizing local and global variance losses to refine attention distribution, leading to improved summarization quality.

Contribution

It proposes a novel attention refinement unit with variance-based supervision at both local and global levels, enhancing the effectiveness of sequence-to-sequence models.

Findings

Improved performance on CNN/Daily Mail dataset

Effective attention distribution optimization

Enhanced saliency and reduced repetitions

Abstract

Attention plays a key role in the improvement of sequence-to-sequence-based document summarization models. To obtain a powerful attention helping with reproducing the most salient information and avoiding repetitions, we augment the vanilla attention model from both local and global aspects. We propose an attention refinement unit paired with local variance loss to impose supervision on the attention model at each decoding step, and a global variance loss to optimize the attention distributions of all decoding steps from the global perspective. The performances on the CNN/Daily Mail dataset verify the effectiveness of our methods.