High-Resolution Modeling of the Fastest First-Order Optimization Method for Strongly Convex Functions

TL;DR

This paper develops a high-resolution ODE model for the accelerated triple momentum (TM) algorithm, revealing its faster convergence compared to Nesterov's method and analyzing its stability and robustness.

Contribution

It introduces a high-resolution ODE representation of the TM algorithm, providing new insights into its convergence rate and robustness compared to existing methods.

Findings

TM ODE model converges faster than NAG ODE model

High-resolution modeling captures TM algorithm characteristics accurately

The analysis shows robustness of the TM ODE to parameter deviations

Abstract

Motivated by the fact that the gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs), here we derive an ODE representation of the accelerated triple momentum (TM) algorithm. For unconstrained optimization problems with strongly convex cost, the TM algorithm has a proven faster convergence rate than the Nesterov's accelerated gradient (NAG) method but with the same computational complexity. We show that similar to the NAG method to capture accurately the characteristics of the TM method, we need to use a high-resolution modeling to obtain the ODE representation of the TM algorithm. We use a Lyapunov analysis to investigate the stability and convergence behavior of the proposed high-resolution ODE representation of the TM algorithm. We show through this analysis that this ODE model has robustness to deviation from the parameters of the TM algorithm. We compare the rate of the ODE representation of the TM method with that of the NAG method to confirm its faster convergence. Our study also leads to a tighter bound on the worst rate of convergence for the ODE model of the NAG method. Lastly, we discuss the use of the integral quadratic constraint (IQC) method to establish an estimate on the rate of convergence of the TM algorithm. A numerical example demonstrates our results.