A Stable High-order Tuner for General Convex Functions

TL;DR

This paper extends the High-order Tuner (HT) algorithm to general convex functions, providing stability and convergence guarantees for dynamic systems and demonstrating accelerated learning through simulations.

Contribution

It generalizes the HT algorithm from linear regression to all convex functions, ensuring stability and convergence in dynamic environments.

Findings

The extended HT guarantees stability for convex functions with time-varying regressors.

The algorithm achieves asymptotic convergence under convexity and smoothness conditions.

Numerical simulations confirm accelerated learning and stability of the proposed method.

Abstract

Iterative gradient-based algorithms have been increasingly applied for the training of a broad variety of machine learning models including large neural-nets. In particular, momentum-based methods, with accelerated learning guarantees, have received a lot of attention due to their provable guarantees of fast learning in certain classes of problems and multiple algorithms have been derived. However, properties for these methods hold only for constant regressors. When time-varying regressors occur, which is commonplace in dynamic systems, many of these momentum-based methods cannot guarantee stability. Recently, a new High-order Tuner (HT) was developed for linear regression problems and shown to have 1) stability and asymptotic convergence for time-varying regressors and 2) non-asymptotic accelerated learning guarantees for constant regressors. In this paper, we extend and discuss the results of this same HT for general convex loss functions. Through the exploitation of convexity and smoothness definitions, we establish similar stability and asymptotic convergence guarantees. Finally, we provide numerical simulations supporting the satisfactory behavior of the HT algorithm as well as an accelerated learning property.