An Adaptive and Parameter-Free Nesterov's Accelerated Gradient Method for Convex Optimization

TL;DR

This paper introduces AdaNAG, an adaptive, parameter-free Nesterov's accelerated gradient method that achieves optimal convergence rates for convex optimization without line-search, along with a new adaptive gradient descent method, AdaGD.

Contribution

The paper presents AdaNAG, a novel adaptive, parameter-free accelerated gradient method with proven convergence rates, and introduces AdaGD, a new adaptive gradient descent method, supported by Lyapunov analysis.

Findings

AdaNAG achieves $O(1/k^2)$ convergence rate.

AdaGD attains $O(1/k)$ convergence rate.

Numerical results show superior performance over recent adaptive methods.

Abstract

We propose AdaNAG, an adaptive accelerated gradient method based on Nesterov's accelerated gradient method. AdaNAG is line-search-free, parameter-free, and achieves the accelerated convergence rates $f(x_k) - f_\star = \mathcal{O}\left(1/k^2\right)$ and $\min_{i\in\left\{1,\dots, k\right\}} \|\nabla f(x_i)\|^2 = \mathcal{O}\left(1/k^3\right)$ for $L$-smooth convex function $f$. We provide a Lyapunov analysis for the convergence proof of AdaNAG, which additionally enables us to propose a novel adaptive gradient descent (GD) method, AdaGD. AdaGD achieves the non-ergodic convergence rate $f(x_k) - f_\star = \mathcal{O}\left(1/k\right)$, like the original GD. The analysis of AdaGD also motivated us to propose a generalized AdaNAG that includes practically useful variants of AdaNAG. Numerical results demonstrate that our methods outperform some other recent adaptive methods for representative applications.