Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise

TL;DR

This paper establishes a worst-case complexity theory for stochastically preconditioned SGD under heavy-tailed noise, showing normalization's advantages over clipping in convergence guarantees and providing insights into empirical training practices.

Contribution

The paper introduces a worst-case complexity framework for SPSGD with heavy-tailed noise, demonstrating normalization's superior convergence properties compared to clipping.

Findings

Normalization guarantees convergence at specific rates under known and unknown parameters.

Clipping may fail to converge in the worst case due to dependence issues.

A novel vector-valued Burkholder-type inequality was developed for analysis.

Abstract

We develop a worst-case complexity theory for stochastically preconditioned stochastic gradient descent (SPSGD) and its accelerated variants under heavy-tailed noise, a setting that encompasses widely used adaptive methods such as Adam, RMSProp, and Shampoo. We assume the stochastic gradient noise has a finite $p$-th moment for some $p \in (1,2]$, and measure convergence after $T$ iterations. While clipping and normalization are parallel tools for stabilizing training of SGD under heavy-tailed noise, there is a fundamental separation in their worst-case properties in stochastically preconditioned settings. We demonstrate that normalization guarantees convergence to a first-order stationary point at rate $\mathcal{O}(T^{-\frac{p-1}{3p-2}})$ when problem parameters are known, and $\mathcal{O}(T^{-\frac{p-1}{2p}})$ when problem parameters are unknown, matching the optimal rates for normalized SGD, respectively. In contrast, we prove that clipping may fail to converge in the worst case due to the statistical dependence between the stochastic preconditioner and the gradient estimates. To enable the analysis, we develop a novel vector-valued Burkholder-type inequality that may be of independent interest. These results provide a theoretical explanation for the empirical preference for normalization over clipping in large-scale model training.