Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models

TL;DR

This paper investigates the role of feedforward networks in transformer models, demonstrating that three-layer FFNs can outperform standard configurations, leading to more efficient language models.

Contribution

It provides empirical evidence that multi-layer FFNs are crucial for transformer performance and introduces a more efficient architecture with three-layer FFNs.

Findings

Three-layer FFNs outperform two-layer FFNs in training loss.

Models with fewer blocks and three-layer FFNs are more parameter-efficient.

Feedforward networks significantly impact transformer model effectiveness.

Abstract

Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we examine the importance of the FFN during the model pre-training process through a series of experiments, confirming that the FFN is important to model performance. Furthermore, we show that models using a transformer block configuration with three-layer FFNs with fewer such blocks outperform the standard two-layer configuration delivering lower training loss with fewer total parameters in less time.