Does Weight Decay Enhance Training Stability?

TL;DR

This paper investigates how weight decay influences training stability in deep learning, revealing architecture-dependent phase transitions and mechanisms that affect parameter dynamics and loss sharpness.

Contribution

It uncovers the effects of weight decay on the Edge of Stability, models the phenomena mathematically, and challenges traditional stability diagnostics based on curvature thresholds.

Findings

Weight decay slows progressive sharpening of the loss landscape.

In CNNs, weight decay dampens oscillations at the Edge of Stability.

In MLPs, increasing weight decay causes a phase transition in sharpness stabilization.

Abstract

In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the \emph{Edge of Stability} (EoS). We show that weight decay robustly slows *progressive sharpening}. Furthermore, we uncover a striking architecture-dependent phase transition. In CNNs, weight decay dampens the oscillations at the EoS, while in MLPs, increasing weight decay causes a phase transition in which the sharpness stabilizes at a threshold significantly below the theoretical $\frac{2}{\eta}$ boundary. We develop a mathematical framework that accurately models these phenomena and identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition. Importantly, we show that these phenomena translate into stability in terms of search in function-space (NTK). Last, this shows that curvature thresholds obtained from convex/quadratic heuristics may not be reliable stability diagnostics under regularization.