Post-LayerNorm Is Back: Stable, ExpressivE, and Deep

TL;DR

This paper demonstrates that a modified Post-LayerNorm Transformer, called Keel, with Highway-style connections, enables stable training of extremely deep language models, surpassing Pre-LN in depth scalability and performance.

Contribution

The authors introduce Keel, a Post-LayerNorm Transformer with Highway connections, allowing stable training of models over 1000 layers without complex tricks.

Findings

Keel trains reliably at depths over 1000 layers.

Keel outperforms Pre-LN in perplexity and depth-scaling.

Post-LN with Highway connections enables extremely deep LLMs.

Abstract

Large language model (LLM) scaling is hitting a wall. Widening models yields diminishing returns, and extending context length does not improve fundamental expressivity. In contrast, depth scaling offers theoretically superior expressivity, yet current Transformer architectures struggle to train reliably at extreme depths. We revisit the Post-LayerNorm (Post-LN) formulation, whose instability at scale caused its replacement by Pre-LN in modern LLMs. We show that the central failure mode of Post-LN arises from the ResNet-style residual pathway, which introduces gradient vanishing in deep networks. We present Keel, a Post-LN Transformer that replaces this residual path with a Highway-style connection. This modification preserves the gradient flow through the residual branch, preventing signal vanishing from the top layers to the bottom. Unlike prior methods, Keel enables stable training at extreme depths without requiring specialized initialization or complex optimization tricks. Keel trains robustly at depths exceeding 1000 layers and consistently improves perplexity and depth-scaling characteristics over Pre-LN. These findings indicate that Post-LN, when paired with a Highway-style connection, provides a simple and effective foundation for building deeply scalable LLMs, opening the possibility for future infinite-depth architectures.