AdaGrad stepsizes: Sharp convergence over nonconvex landscapes

TL;DR

This paper provides the first theoretical convergence guarantees for AdaGrad in nonconvex optimization, demonstrating sharp rates and robustness to hyper-parameter choices, supported by extensive experiments.

Contribution

It establishes convergence rates for AdaGrad-Norm on nonconvex functions, bridging a theoretical gap and showing robustness compared to standard SGD.

Findings

AdaGrad-Norm converges at 0(\,rac{ log(N)}{\, ext{N}}) rate in stochastic settings.

Converges at 0(\, ext{N}) rate in batch (non-stochastic) setting.

Robustness to hyper-parameters extends to deep learning models.

Abstract

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule. Yet, the theoretical guarantees to date for AdaGrad are for online and convex optimization. We bridge this gap by providing theoretical guarantees for the convergence of AdaGrad for smooth, nonconvex functions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges to a stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in the stochastic setting, and at the optimal $\mathcal{O}(1/N)$ rate in the batch (non-stochastic) setting -- in this sense, our convergence guarantees are 'sharp'. In particular, the convergence of AdaGrad-Norm is robust to the choice of all hyper-parameters of the algorithm, in contrast to stochastic gradient descent whose convergence depends crucially on tuning the step-size to the (generally unknown) Lipschitz smoothness constant and level of stochastic noise on the gradient. Extensive numerical experiments are provided to corroborate our theory; moreover, the experiments suggest that the robustness of AdaGrad-Norm extends to state-of-the-art models in deep learning, without sacrificing generalization.