Revisiting the acceleration phenomenon via high-resolution differential equations

TL;DR

This paper deepens understanding of Nesterov's accelerated gradient descent by analyzing high-resolution differential equations, simplifying proofs, enlarging step sizes, and comparing explicit and implicit schemes for better convergence insights.

Contribution

It introduces a simplified Lyapunov analysis for NAG using high-resolution differential equations, enlarges step sizes, and compares explicit and implicit schemes for improved convergence understanding.

Findings

Simplified proof of NAG convergence using gradient-correction scheme.

Enlarged step size to 1/L with minor modifications.

Implicit-velocity scheme yields a more effective high-resolution differential equation framework.

Abstract

Nesterov's accelerated gradient descent (NAG) is one of the milestones in the history of first-order algorithms. It was not successfully uncovered until the high-resolution differential equation framework was proposed in [Shi et al., 2022] that the mechanism behind the acceleration phenomenon is due to the gradient correction term. To deepen our understanding of the high-resolution differential equation framework on the convergence rate, we continue to investigate NAG for the $\mu$-strongly convex function based on the techniques of Lyapunov analysis and phase-space representation in this paper. First, we revisit the proof from the gradient-correction scheme. Similar to [Chen et al., 2022], the straightforward calculation simplifies the proof extremely and enlarges the step size to $s=1/L$ with minor modification. Meanwhile, the way of constructing Lyapunov functions is principled. Furthermore, we also investigate NAG from the implicit-velocity scheme. Due to the difference in the velocity iterates, we find that the Lyapunov function is constructed from the implicit-velocity scheme without the additional term and the calculation of iterative difference becomes simpler. Together with the optimal step size obtained, the high-resolution differential equation framework from the implicit-velocity scheme of NAG is perfect and outperforms the gradient-correction scheme.