Synthesizer: Rethinking Self-Attention in Transformer Models

TL;DR

This paper questions the necessity of dot product self-attention in Transformers, introducing Synthesizer models that learn synthetic attention weights, achieving competitive or superior performance across various NLP tasks.

Contribution

The paper proposes Synthesizer, a novel attention mechanism that forgoes token-token interactions, demonstrating competitive performance and efficiency gains over traditional Transformers.

Findings

Random Synthesizer performs surprisingly well.

Synthesizers outperform Transformers when combined with dot product attention.

Simple Synthesizers are faster and more effective than Dynamic Convolutions.

Abstract

The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.