Stronger Normalization-Free Transformers

TL;DR

This paper introduces Derf, a novel point-wise function that surpasses existing normalization techniques like LayerNorm, RMSNorm, and DyT in various domains, simplifying Transformer architectures.

Contribution

The work presents a large-scale search for effective point-wise functions, identifying Derf as superior to previous normalization methods in diverse tasks.

Findings

Derf outperforms LayerNorm, RMSNorm, and DyT across multiple domains.

Performance gains are mainly due to improved generalization, not fitting capacity.

Derf's simplicity makes it a practical normalization-free alternative for Transformers.

Abstract

Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce $\mathrm{Derf}(x) = \mathrm{erf}(\alpha x + s)$, where $\mathrm{erf}(x)$ is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation, speech representation, and DNA sequence modeling. Our analysis also suggests that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.