SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance

TL;DR

This paper provides a comprehensive high-probability analysis of AdaGrad stochastic gradient descent, demonstrating its full adaptivity to unknown parameters, unbounded gradients, and affine variance noise in both convex and non-convex settings.

Contribution

It offers the first analysis of AdaGrad that removes prior limitations, supporting a general noise model and providing sharp convergence rates without assuming problem parameter knowledge.

Findings

Supports a general affine variance noise model

Provides sharp convergence rates in low and high noise regimes

Achieves high-probability bounds without strong global assumptions

Abstract

We study Stochastic Gradient Descent with AdaGrad stepsizes: a popular adaptive (self-tuning) method for first-order stochastic optimization. Despite being well studied, existing analyses of this method suffer from various shortcomings: they either assume some knowledge of the problem parameters, impose strong global Lipschitz conditions, or fail to give bounds that hold with high probability. We provide a comprehensive analysis of this basic method without any of these limitations, in both the convex and non-convex (smooth) cases, that additionally supports a general ``affine variance'' noise model and provides sharp rates of convergence in both the low-noise and high-noise~regimes.