Pay Attention to MLPs

TL;DR

This paper introduces gMLP, a simple MLP-based architecture with gating, demonstrating it can match or surpass Transformers in language and vision tasks, challenging the notion that self-attention is essential.

Contribution

The paper presents gMLP, a novel MLP-based model that achieves competitive performance with Transformers in various applications, showing self-attention is not always necessary.

Findings

gMLP performs on par with Transformers in vision and NLP tasks.

Self-attention is not critical for Vision Transformers.

Scaling gMLP can close performance gaps with Transformers.

Abstract

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.