The Nesterov-Spokoiny Acceleration Achieves Strict $o(1/k^2)$ Convergence

TL;DR

This paper demonstrates that the Nesterov-Spokoiny Acceleration method achieves a strict convergence rate faster than previous bounds in smooth convex optimization, with extensions to inexact gradients and composite problems.

Contribution

It introduces a strict $o(1/k^2)$ convergence rate for NSA, extends it to zeroth-order and composite optimization, and provides a continuous-time analysis linking to high-resolution ODEs.

Findings

NSA achieves strict $o(1/k^2)$ convergence in function value.

NSA extends to inexact gradients with similar convergence.

Continuous-time analysis connects NSA to high-resolution ODEs.

Abstract

This paper studies the Nesterov-Spokoiny Acceleration (NSA), a variant of the accelerated gradient method by Nesterov and Spokoiny. For smooth convex optimization, NSA achieves a strict $o(1/k^2)$ convergence rate in function value and an $o(1/(k^3 \log k))$ rate in squared gradient norm, while ensuring monotonic descent of the objective. We further study a zeroth-order version of NSA that handles inexact gradients, and extends NSA to composite optimization problems, in each case establishing $o(1/k^2)$ convergence in function value. A continuous-time analysis reveals connections to high-resolution ODEs known to underlie acceleration phenomena.