Acceleration via Symplectic Discretization of High-Resolution Differential Equations

TL;DR

This paper investigates discretization schemes for ODEs related to Nesterov's and Polyak's methods, demonstrating that symplectic discretization of a high-resolution ODE achieves accelerated convergence in smooth strongly convex optimization.

Contribution

It introduces a symplectic discretization approach for high-resolution ODEs that yields accelerated optimization algorithms, highlighting the importance of scheme choice for practical acceleration.

Findings

Symplectic discretization achieves acceleration for smooth strongly convex functions.

Explicit Euler scheme may not achieve acceleration or be practical.

Implicit schemes are often impractical or fail to accelerate.

Abstract

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme. We show that the optimization algorithm generated by applying the symplectic scheme to a high-resolution ODE proposed by Shi et al. [2018] achieves an accelerated rate for minimizing smooth strongly convex functions. On the other hand, the resulting algorithm either fails to achieve acceleration or is impractical when the scheme is implicit, the ODE is low-resolution, or the scheme is explicit.