A Continuized View on Nesterov Acceleration

TL;DR

This paper introduces a novel 'continuized' version of Nesterov acceleration, blending continuous differential equations with discrete gradient steps, enabling precise analysis and maintaining optimal convergence rates.

Contribution

It presents a new continuized framework for Nesterov acceleration, combining continuous analysis with discrete implementation, and introduces a random-parameter discretization method.

Findings

The continuized approach allows differential calculus analysis of convergence.

Discretization of the continuized process retains Nesterov's convergence rates.

The method bridges continuous and discrete optimization frameworks.

Abstract

We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradient steps at random times. This continuized variant benefits from the best of the continuous and the discrete frameworks: as a continuous process, one can use differential calculus to analyze convergence and obtain analytical expressions for the parameters; but a discretization of the continuized process can be computed exactly with convergence rates similar to those of Nesterov original acceleration. We show that the discretization has the same structure as Nesterov acceleration, but with random parameters.