Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures

TL;DR

This paper introduces a method to enhance sparse transformer models by selectively increasing global attention on key keywords, improving long-range context encoding for abstractive summarization tasks across various datasets.

Contribution

It proposes a keyword-based global attention mechanism extension to Longformer, improving long-range context encoding in sparse transformers for summarization.

Findings

Improved zero-shot and few-shot summarization performance.

Enhanced encoding of long-range dependencies.

Demonstrated effectiveness on multiple benchmark datasets.

Abstract

In this paper, we propose an extension to Longformer Encoder-Decoder, a popular sparse transformer architecture. One common challenge with sparse transformers is that they can struggle with encoding of long range context, such as connections between topics discussed at a beginning and end of a document. A method to selectively increase global attention is proposed and demonstrated for abstractive summarization tasks on several benchmark data sets. By prefixing the transcript with additional keywords and encoding global attention on these keywords, improvement in zero-shot, few-shot, and fine-tuned cases is demonstrated for some benchmark data sets.