Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methods

TL;DR

This paper introduces a unified continuous-time modeling framework for Nesterov's accelerated gradient methods, enabling comprehensive analysis, convergence rate determination, and the development of a generalized restart scheme with broad applicability.

Contribution

It presents a generalized continuous-time model that unifies existing models, determines convergence rates, and proposes a new restart scheme applicable to a wider class of Nesterov's methods.

Findings

Six existing models are special cases of the generalized model.

The generalized models achieve accelerated convergence rates via time reparametrization.

The proposed restart scheme ensures monotonic decrease in objective function values.

Abstract

Recent research has indicated a substantial rise in interest in understanding Nesterov's accelerated gradient methods via their continuous-time models. However, most existing studies focus on specific classes of Nesterov's methods, which hinders the attainment of an in-depth understanding and a unified perspective. To address this deficit, we present generalized continuous-time models that cover a broad range of Nesterov's methods, including those previously studied under existing continuous-time frameworks. Our key contributions are as follows. First, we identify the convergence rates of the generalized models, eliminating the need to determine the convergence rate for any specific continuous-time model derived from them. Second, we show that six existing continuous-time models are special cases of our generalized models, thereby positioning our framework as a unifying tool for analyzing and understanding these models. Third, we design a restart scheme for Nesterov's methods based on our generalized models and show that it ensures a monotonic decrease in objective function values. Owing to the broad applicability of our models, this scheme can be used to a broader class of Nesterov's methods compared to the original restart scheme. Fourth, we uncover a connection between our generalized models and gradient flow in continuous time, showing that the accelerated convergence rates of our generalized models can be attributed to a time reparametrization in gradient flow. Numerical experiment results are provided to support our theoretical analyses and results.