Accelerated Gradient Methods via Inertial Systems with Hessian-driven Damping

TL;DR

This paper studies the convergence rates of inertial algorithms derived from Hessian-driven damping systems, showing potential speedups over Nesterov's method and providing explicit parameter dependence insights.

Contribution

It introduces a family of inertial algorithms with proven accelerated convergence rates and explicit parameter dependence, enhancing understanding of acceleration mechanisms.

Findings

Achieves up to 2x speedup over Nesterov's scheme for smooth strongly convex functions.

Establishes linear convergence for functions with quadratic growth or Polyak-jasiewicz inequality.

Provides explicit relations between convergence rates and algorithm parameters.

Abstract

We analyze the convergence rate of a family of inertial algorithms, which can be obtained by discretization of an inertial system with Hessian-driven damping. We recover a convergence rate, up to a factor of 2 speedup upon Nesterov's scheme, for smooth strongly convex functions. As a byproduct of our analyses, we also derive linear convergence rates for convex functions satisfying quadratic growth condition or Polyak-\L ojasiewicz inequality. As a significant feature of our results, the dependence of the convergence rate on parameters of the inertial system/algorithm is revealed explicitly. This may help one get a better understanding of the acceleration mechanism underlying an inertial algorithm.