Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos

TL;DR

This paper investigates the dynamics of sharpness in neural network training, revealing mechanisms behind stability, sharpness trends, and chaos, using a simplified linear model that generalizes to real-world training.

Contribution

The paper introduces a simplified 2-layer linear model that captures key sharpness phenomena in neural training and analyzes the underlying mechanisms and conditions for stability and chaos.

Findings

Early sharpness reduction and progressive sharpening explained

Conditions for edge of stability identified

Period-doubling route to chaos discovered

Abstract

In gradient descent dynamics of neural networks, the top eigenvalue of the loss Hessian (sharpness) displays a variety of robust phenomena throughout training. This includes early time regimes where the sharpness may decrease during early periods of training (sharpness reduction), and later time behavior such as progressive sharpening and edge of stability. We demonstrate that a simple $2$-layer linear network (UV model) trained on a single training example exhibits all of the essential sharpness phenomenology observed in real-world scenarios. By analyzing the structure of dynamical fixed points in function space and the vector field of function updates, we uncover the underlying mechanisms behind these sharpness trends. Our analysis reveals (i) the mechanism behind early sharpness reduction and progressive sharpening, (ii) the required conditions for edge of stability, (iii) the crucial role of initialization and parameterization, and (iv) a period-doubling route to chaos on the edge of stability manifold as learning rate is increased. Finally, we demonstrate that various predictions from this simplified model generalize to real-world scenarios and discuss its limitations.