Stability of first-order methods in tame optimization

TL;DR

This paper investigates the stability of first-order optimization methods with constant step size when applied to locally Lipschitz tame functions, providing new stability criteria and insights into local minima behavior.

Contribution

It introduces notions of discrete Lyapunov stability for first-order methods and characterizes their stability conditions in non-convex, irregular optimization problems.

Findings

Necessary and sufficient conditions for stability of first-order methods.

Certain local minima can be unstable without added noise.

Connection established between discrete iterates and continuous-time dynamics.

Abstract

Modern data science applications demand solving large-scale optimization problems. The prevalent approaches are first-order methods, valued for their scalability. These methods are implemented to tackle highly irregular problems where assumptions of convexity and smoothness are untenable. Seeking to deepen the understanding of these methods, we study first-order methods with constant step size for minimizing locally Lipschitz tame functions. To do so, we propose notions of discrete Lyapunov stability for optimization methods. Concerning common first-order methods, we provide necessary and sufficient conditions for stability. We also show that certain local minima can be unstable, without additional noise in the method. Our analysis relies on the connection between the iterates of the first-order methods and continuous-time dynamics.