Lost in Backpropagation: The LM Head is a Gradient Bottleneck

TL;DR

The paper reveals that the last layer of neural language models creates a gradient bottleneck due to the size mismatch between features and vocabulary, impairing training efficiency and expressivity.

Contribution

It provides a theoretical and empirical analysis of the gradient bottleneck caused by the LM head, highlighting its impact on training dynamics and proposing the need for new head designs.

Findings

95-99% of gradient norm is suppressed by the output layer

Gradient compression alters training feedback and updates

Gradient bottleneck hampers learning of trivial patterns

Abstract

The last layer of neural language models (LMs) projects output features of dimension $D$ to logits in dimension $V$, the size of the vocabulary, where usually $D \ll V$. This mismatch is known to raise risks of limited expressivity in neural LMs, creating a so-called softmax bottleneck. We show the softmax bottleneck is not only an expressivity bottleneck but also an optimization bottleneck. Backpropagating $V$-dimensional gradients through a rank-$D$ linear layer induces unavoidable compression, which alters the training feedback provided to the vast majority of the parameters. We present a theoretical analysis of this phenomenon and measure empirically that 95-99% of the gradient norm is suppressed by the output layer, resulting in vastly suboptimal update directions. We conduct controlled pretraining experiments showing that the gradient bottleneck makes trivial patterns unlearnable, and drastically affects the training dynamics of LLMs. We argue that this inherent flaw contributes to training inefficiencies at scale independently of the model architecture, and raises the need for new LM head designs.