A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

TL;DR

This paper challenges the Gaussian assumption of gradient noise in SGD for deep learning, proposing a heavy-tailed alpha-stable model driven by Lévy motion, supported by extensive empirical evidence across architectures and datasets.

Contribution

It introduces a non-Gaussian heavy-tailed model for gradient noise in SGD, replacing the classical Brownian motion framework with Lévy motion analysis.

Findings

Gradient noise is highly non-Gaussian with heavy tails across settings.

SGD transitions from narrow to wide minima due to jumps in Lévy-driven SDEs.

Heavy-tailed behavior varies with architecture, size, and dataset.

Abstract

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed $\alpha$-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a L\'{e}vy motion. Such SDEs can incur `jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the $\alpha$-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.