Attention Is Not All You Need Anymore
Zhe Chen

TL;DR
This paper introduces a family of replacement modules called Extractors for the Transformer's self-attention, improving performance and efficiency, and demonstrates their effectiveness through experiments and theoretical analysis.
Contribution
Proposes novel Extractor modules as drop-in replacements for self-attention, enhancing Transformer performance and reducing computational complexity.
Findings
SHE improves Transformer performance significantly.
Simplified Extractors match or outperform self-attention with less complexity.
Extractors can run faster due to shorter critical computation paths.
Abstract
In recent years, the popular Transformer architecture has achieved great success in many application areas, including natural language processing and computer vision. Many existing works aim to reduce the computational and memory complexity of the self-attention mechanism in the Transformer by trading off performance. However, performance is key for the continuing success of the Transformer. In this paper, a family of drop-in replacements for the self-attention mechanism in the Transformer, called the Extractors, is proposed. Four types of the Extractors, namely the super high-performance Extractor (SHE), the higher-performance Extractor (HE), the worthwhile Extractor (WE), and the minimalist Extractor (ME), are proposed as examples. Experimental results show that replacing the self-attention mechanism with the SHE evidently improves the performance of the Transformer, whereas the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Web Data Mining and Analysis · Natural Language Processing Techniques
MethodsMulti-Head Attention · Attention Is All You Need · Adam · Softmax · Label Smoothing · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Layer Normalization · Linear Layer · Residual Connection
