Adaptive Acceleration Without Strong Convexity Priors Or Restarts

TL;DR

This paper introduces NAG-free, an extension of Nesterov's accelerated gradient method that can estimate the strong convexity parameter without prior knowledge or restarts, achieving adaptive acceleration.

Contribution

It presents the first method capable of estimating the strong convexity parameter directly without priors or restarts, and demonstrates its theoretical and practical effectiveness.

Findings

NAG-free converges globally at least as fast as gradient descent.

It achieves accelerated convergence locally near the minimum.

Experiments show NAG-free is competitive and adapts to local curvature.

Abstract

A longstanding challenge in optimization is achieving optimal performance when the strong convexity parameter m is unknown. In this paper, we propose NAG-free, a simple extension of Nesterov's accelerated gradient (NAG) which is the first method capable of estimating m directly, without priors or restarts. Our estimator is inexpensive: it requires no additional function or gradient evaluations, only the storage of one extra iterate and gradient already computed by NAG. We prove that, by estimating the smoothness parameter L via backtracking, NAG-free converges globally at least as fast as gradient descent. We also prove that, given an upper bound on L, NAG-free achieves accelerated convergence locally near the minimum under local smoothness of the Hessian and some mild additional assumptions. Finally, we present experiments with smooth and nonsmooth Hessians on both synthetic and real-world data which demonstrate that NAG-free is competitive with restart methods, and naturally adapts to favorable local curvature conditions.