An Abstract Lyapunov Control Optimizer: Local Stabilization and Global Convergence

TL;DR

This paper introduces a novel Lyapunov control framework for adaptive optimization algorithms, providing stability and convergence guarantees by leveraging continuous dynamical systems and topology-based selection theory.

Contribution

It develops a new theoretical framework for analyzing adaptive optimizers using Lyapunov stability and continuous selection theory, extending results to various momentum and gradient descent methods.

Findings

Proves stability and convergence of adaptive optimizers with arbitrary hyperparameters.

Introduces a novel use of continuous selection theory in optimization analysis.

Provides new insights into momentum and p-GD algorithms with adaptive step sizes.

Abstract

Recently, many machine learning optimizers have been analysed considering them as the asymptotic limit of some differential equations when the step size goes to zero. In other words, the optimizers can be seen as a finite difference scheme applied to a continuous dynamical system. But the major part of the results in the literature concerns constant step size algorithms. The main aim of this paper is to investigate the guarantees of the adaptive step size counterpart. In fact, this dynamical point of view can be used to design step size update rules, by choosing a discretization of the continuous equation that preserves its most relevant features. In this work, we analyse this kind of adaptive optimizers and prove their Lyapunov stability and convergence properties for any choice of hyperparameters. At the best of our knowledge, this paper introduces for the first time the use of continuous selection theory from general topology to overcome some of the intrinsic difficulties due to the non constant and non regular step size policies. The general framework developed gives many new results on adaptive and constant step size Momentum/Heavy-Ball and p-GD algorithms.