A Note on the Gradient-Evaluation Sequence in Accelerated Gradient Methods

TL;DR

This paper investigates the gradient-evaluation sequence in accelerated gradient methods, proving it can achieve optimal iteration complexity for convex smooth problems, including constrained and non-Euclidean cases.

Contribution

It establishes that the gradient-evaluation sequence in AGD attains the same optimal complexity as the approximate solution sequence, extending results to constrained and non-Euclidean settings.

Findings

Gradient-evaluation sequence achieves O(L/k^2) convergence.

Results hold for constrained and non-Euclidean settings.

Confirms optimal complexity for gradient-evaluation sequence in AGD.

Abstract

Nesterov's accelerated gradient descent method (AGD) is a seminal deterministic first-order method known to achieve the optimal order of iteration complexity for solving convex smooth optimization problems. Two distinct sequences of iterates are included in the description of AGD: gradient evaluations are performed at one sequence, while approximate solutions are selected from the other. The iteration complexity on minimizing objective function value has been well-studied in the literature, but such analysis is almost always performed only at the approximate solution sequence. To the best of our knowledge, for projection-based AGD that solves problems with feasible sets, it is still an open research question whether the gradient evaluation sequence (when treated as approximate solutions) could also achieve the same optimal order of iteration complexity. It is also unknown whether such results still hold in the non-Euclidean setting. Motivated by computer-aided algorithm analysis, we provide positive results that answer the open problems affirmatively. Specifically, for (possibly constrained) problem $f^*:=\min_{x\in X}f(x)$ where $f$ is convex and $L$-smooth and $X$ is closed, convex and projection friendly, we prove that the gradient-evaluation sequence $\{\underline{x}_k\}$ in AGD satisfies that $f(\underline{x}_k) - f^*\le \mathcal O(L/k^2)$.