Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness

TL;DR

This paper introduces a generalized smoothness framework for analyzing nonlinearly preconditioned gradient methods, extending existing theories to broader classes of functions and providing convergence insights for both convex and nonconvex problems.

Contribution

It develops a new generalized smoothness property based on abstract convexity, encompassing algorithms like gradient clipping and extending analysis to $(L_0,L_1)$-smooth functions.

Findings

Framework encapsulates gradient clipping algorithms.

Provides convergence analysis for convex and nonconvex cases.

Extends beyond traditional Lipschitz smoothness assumptions.

Abstract

We analyze nonlinearly preconditioned gradient methods for solving smooth minimization problems. We introduce a generalized smoothness property, based on the notion of abstract convexity, that is broader than Lipschitz smoothness and provide sufficient first- and second-order conditions. Notably, our framework encapsulates algorithms associated with the gradient clipping method and brings out novel insights for the class of $(L_0,L_1)$-smooth functions that has received widespread interest recently, thus allowing us to extend beyond already established methods. We investigate the convergence of the proposed method in both the convex and nonconvex setting.