The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control

TL;DR

This paper provides a theoretical explanation for how normalization layers in neural networks control capacity exponentially, leading to improved training stability and generalization, which explains their empirical success.

Contribution

It introduces a capacity control framework showing normalization layers exponentially reduce Lipschitz constants, explaining their role in optimization and generalization.

Findings

Normalization layers exponentially reduce Lipschitz constants.

They smooth the loss landscape exponentially.

They constrain network capacity, improving generalization.

Abstract

Normalization layers are critical components of modern AI systems, such as ChatGPT, Gemini, DeepSeek, etc. Empirically, they are known to stabilize training dynamics and improve generalization ability. However, the underlying theoretical mechanism by which normalization layers contribute to both optimization and generalization remains largely unexplained, especially when using many normalization layers in a deep neural network (DNN). In this work, we develop a theoretical framework that elucidates the role of normalization through the lens of capacity control. We prove that an unnormalized DNN can exhibit exponentially large Lipschitz constants with respect to either its parameters or inputs, implying excessive functional capacity and potential overfitting. Such bad DNNs are uncountably many. In contrast, the insertion of normalization layers provably can reduce the Lipschitz constant at an exponential rate in the number of normalization layers. This exponential reduction yields two fundamental consequences: (1) it smooths the loss landscape at an exponential rate, facilitating faster and more stable optimization; and (2) it constrains the effective capacity of the network, thereby enhancing generalization guarantees on unseen data. Our results thus offer a principled explanation for the empirical success of normalization methods in deep learning.