On the stability of gradient descent with second order dynamics for time-varying cost functions

TL;DR

This paper investigates the stability of gradient descent algorithms with second order dynamics in the context of time-varying cost functions, providing broader stability guarantees to enhance safety in real-time machine learning applications.

Contribution

It extends previous stability results for second order gradient descent methods to more general time-varying scenarios, aiding in their safe deployment.

Findings

Provides generalized stability guarantees for second order gradient descent.

Facilitates design and certification of optimization algorithms for real-time systems.

Encourages cross-disciplinary analysis between online learning and stochastic optimization.

Abstract

Gradient based optimization algorithms deployed in Machine Learning (ML) applications are often analyzed and compared by their convergence rates or regret bounds. While these rates and bounds convey valuable information they don't always directly translate to stability guarantees. Stability and similar concepts, like robustness, will become ever more important as we move towards deploying models in real-time and safety critical systems. In this work we build upon the results in Gaudio et al. 2021 and Moreu & Annaswamy 2022 for gradient descent with second order dynamics when applied to explicitly time varying cost functions and provide more general stability guarantees. These more general results can aid in the design and certification of these optimization schemes so as to help ensure safe and reliable deployment for real-time learning applications. We also hope that the techniques provided here will stimulate and cross-fertilize the analysis that occurs on the same algorithms from the online learning and stochastic optimization communities.