Implicit Gradient Regularization

TL;DR

This paper introduces Implicit Gradient Regularization (IGR), a phenomenon where gradient descent implicitly biases models toward flat minima, improving generalization, and demonstrates how to quantify and control this effect using backward error analysis.

Contribution

The paper formalizes IGR, shows its bias toward flat minima, and proposes using it as an explicit regularizer, advancing understanding of gradient descent's implicit biases.

Findings

IGR biases models toward flat minima with small test errors.

Backward error analysis quantifies the size of IGR.

Explicit regularization based on IGR improves control over model training.

Abstract

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.