Attention Is Not All You Need Anymore

TL;DR

This paper introduces a family of replacement modules called Extractors for the Transformer's self-attention, improving performance and efficiency, and demonstrates their effectiveness through experiments and theoretical analysis.

Contribution

Proposes novel Extractor modules as drop-in replacements for self-attention, enhancing Transformer performance and reducing computational complexity.

Findings

SHE improves Transformer performance significantly.

Simplified Extractors match or outperform self-attention with less complexity.

Extractors can run faster due to shorter critical computation paths.

Abstract

In recent years, the popular Transformer architecture has achieved great success in many application areas, including natural language processing and computer vision. Many existing works aim to reduce the computational and memory complexity of the self-attention mechanism in the Transformer by trading off performance. However, performance is key for the continuing success of the Transformer. In this paper, a family of drop-in replacements for the self-attention mechanism in the Transformer, called the Extractors, is proposed. Four types of the Extractors, namely the super high-performance Extractor (SHE), the higher-performance Extractor (HE), the worthwhile Extractor (WE), and the minimalist Extractor (ME), are proposed as examples. Experimental results show that replacing the self-attention mechanism with the SHE evidently improves the performance of the Transformer, whereas the simplified versions of the SHE, i.e., the HE, the WE, and the ME, perform close to or better than the self-attention mechanism with less computational and memory complexity. Furthermore, the proposed Extractors have the potential or are able to run faster than the self-attention mechanism since their critical paths of computation are much shorter. Additionally, the sequence prediction problem in the context of text generation is formulated using variable-length discrete-time Markov chains, and the Transformer is reviewed based on our understanding.