Implicit regularization of normalized gradient descent

TL;DR

This paper investigates how normalized gradient descent with carefully chosen step sizes implicitly favors flat minima, using advanced nonsmooth analysis techniques to understand the underlying regularization effects.

Contribution

It introduces a novel perspective on implicit regularization through normalized gradient descent and applies variational analysis to explain its behavior.

Findings

Normalized gradient descent promotes flat minima under certain conditions

Implicit regularization depends on the existence of Lyapunov functions in gradient dynamics

Nonsmooth analysis provides a framework to understand regularization effects

Abstract

How to find flat minima? We propose running normalized gradient descent, usually reserved for nonsmooth optimization, with sufficiently slowly diminishing step sizes. This induces implicit regularization towards flat minima if an appropriate Lyapunov functions exists in the gradient dynamics. Our analysis shows that implicit regularization is intrinsically a question of nonsmooth analysis, for which we deploy the full power of variational analysis and stratification theory.