Global-aware Beam Search for Neural Abstractive Summarization

TL;DR

This paper introduces a global-aware beam search algorithm for neural abstractive summarization that leverages attention distribution predictions to improve summary quality and robustness across multiple datasets.

Contribution

It proposes a novel global attention-based protocol and scoring mechanism that enhance beam search to achieve near-global optimal summaries in neural summarization.

Findings

Significant improvements over state-of-the-art models on nine datasets.

Robustness to corrupted attention distributions.

Effective in generating meaningful summaries with empirical hyper-parameters.

Abstract

This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available.