On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

TL;DR

This paper reveals that the relationship between SGD noise covariance and loss landscape curvature is more complex than previously assumed, showing an approximate power-law relation rather than direct proportionality, validated across various deep learning models.

Contribution

It introduces a general relationship between SGD noise covariance and per-sample Hessians, challenging the assumption of direct proportionality to the Hessian in deep neural networks.

Findings

SGD noise covariance approximately commutes with the Hessian.

The diagonal elements follow a power-law relation with exponent between 1 and 2.

Experimental validation across datasets and architectures supports the theoretical bounds.

Abstract

Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima. Prior work often assumes an equivalence between the Fisher Information Matrix and the Hessian for negative log-likelihood losses, leading to the claim that the SGD noise covariance $\mathbf{C}$ is proportional to the Hessian $\mathbf{H}$. We show that this assumption holds only under restrictive conditions that are typically violated in deep neural networks. Using the recently discovered Activity--Weight Duality, we find a more general relationship agnostic to the specific loss formulation, showing that $\mathbf{C} \propto \mathbb{E}_p[\mathbf{h}_p^2]$, where $\mathbf{h}_p$ denotes the per-sample Hessian with $\mathbf{H} = \mathbb{E}_p[\mathbf{h}_p]$. As a consequence, $\mathbf{C}$ and $\mathbf{H}$ commute approximately rather than coincide exactly, and their diagonal elements follow an approximate power-law relation $C_{ii} \propto H_{ii}^{\gamma}$ with a theoretically bounded exponent $1 \leq \gamma \leq 2$, determined by per-sample Hessian spectra. Experiments across datasets, architectures, and loss functions validate these bounds, providing a unified characterization of the noise-curvature relationship in deep learning.