Transformers without Tears: Improving the Normalization of Self-Attention

TL;DR

This paper explores normalization techniques in Transformer models, demonstrating that specific normalization strategies improve training stability, efficiency, and translation performance across low-resource and high-resource language tasks.

Contribution

The paper introduces and evaluates three normalization-centric modifications—PreNorm residuals, ScaleNorm, and FixNorm—that enhance Transformer training and translation quality.

Findings

PreNorm and smaller initializations enable warmup-free training with large learning rates.

ScaleNorm accelerates training and improves translation performance.

Normalizing word embeddings to fixed length benefits low-resource translation tasks.

Abstract

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT'15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT'14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance.