Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization

TL;DR

This paper investigates how the interplay between norm minimization and sharpness regularization influences the generalization of neural networks trained with large learning rates, revealing a dynamic trade-off beyond single implicit biases.

Contribution

It demonstrates that implicit regularization involves a balance between norm and sharpness, and that neither bias alone explains generalization, highlighting the need for a broader perspective.

Findings

Learning rate influences the trade-off between norm and sharpness.

Neither implicit bias alone suffices to explain generalization.

Empirical and theoretical analysis of diagonal linear networks supports the trade-off concept.

Abstract

A widely believed explanation for the remarkable generalization capacities of overparameterized neural networks is that the optimization algorithms used for training induce an implicit bias towards benign solutions. To grasp this theoretically, recent works examine gradient descent and its variants in simplified training settings, often assuming vanishing learning rates. These studies reveal various forms of implicit regularization, such as $\ell_1$-norm minimizing parameters in regression and max-margin solutions in classification. Concurrently, empirical findings show that moderate to large learning rates exceeding standard stability thresholds lead to faster, albeit oscillatory, convergence in the so-called Edge-of-Stability regime, and induce an implicit bias towards minima of low sharpness (norm of training loss Hessian). In this work, we argue that a comprehensive understanding of the generalization performance of gradient descent requires analyzing the interaction between these various forms of implicit regularization. We empirically demonstrate that the learning rate balances between low parameter norm and low sharpness of the trained model. We furthermore prove for diagonal linear networks trained on a simple regression task that neither implicit bias alone minimizes the generalization error. These findings demonstrate that focusing on a single implicit bias is insufficient to explain good generalization, and they motivate a broader view of implicit regularization that captures the dynamic trade-off between norm and sharpness induced by non-negligible learning rates.