Gradient Norm Minimization of Nesterov Acceleration: $o(1/k^3)$

TL;DR

This paper investigates the acceleration mechanism of Nesterov's gradient descent, introduces a new high-resolution differential equation framework, and proves that gradient norm minimization can achieve a rate faster than $o(1/k^3)$.

Contribution

It provides a simplified proof of NAG's acceleration, introduces an implicit-velocity high-resolution framework, and establishes a faster convergence rate for gradient norm minimization.

Findings

Proves gradient norm minimization of NAG can achieve $o(1/k^3)$ rate.

Introduces a new implicit-velocity high-resolution differential equation framework.

Shows a faster $o(1/k^2)$ rate for objective value minimization when $r > 2$.

Abstract

In the history of first-order algorithms, Nesterov's accelerated gradient descent (NAG) is one of the milestones. However, the cause of the acceleration has been a mystery for a long time. It has not been revealed with the existence of gradient correction until the high-resolution differential equation framework proposed in [Shi et al., 2021]. In this paper, we continue to investigate the acceleration phenomenon. First, we provide a significantly simplified proof based on precise observation and a tighter inequality for $L$-smooth functions. Then, a new implicit-velocity high-resolution differential equation framework, as well as the corresponding implicit-velocity version of phase-space representation and Lyapunov function, is proposed to investigate the convergence behavior of the iterative sequence $\{x_k\}_{k=0}^{\infty}$ of NAG. Furthermore, from two kinds of phase-space representations, we find that the role played by gradient correction is equivalent to that by velocity included implicitly in the gradient, where the only difference comes from the iterative sequence $\{y_{k}\}_{k=0}^{\infty}$ replaced by $\{x_k\}_{k=0}^{\infty}$. Finally, for the open question of whether the gradient norm minimization of NAG has a faster rate $o(1/k^3)$, we figure out a positive answer with its proof. Meanwhile, a faster rate of objective value minimization $o(1/k^2)$ is shown for the case $r > 2$.