Universal Gradient Methods for Stochastic Convex Optimization

TL;DR

This paper introduces universal gradient algorithms for stochastic convex optimization that adapt to noise and smoothness levels without prior knowledge, achieving state-of-the-art convergence guarantees.

Contribution

The paper presents novel universal gradient methods that automatically adapt to noise and smoothness in stochastic convex optimization, with improved convergence guarantees.

Findings

Achieves state-of-the-art worst-case convergence rates.

Adapts to H"older smoothness without prior knowledge.

Provides optimal efficiency estimates for the universal fast gradient method.

Abstract

We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the H\"older smoothness of the objective function without a priori knowledge of the particular setting. The key ingredient is a novel strategy for adjusting step-size coefficients in the Stochastic Gradient Method (SGD). Unlike AdaGrad, which accumulates gradient norms, our Universal Gradient Method accumulates appropriate combinations of gradient- and iterate differences. The resulting algorithm has state-of-the-art worst-case convergence rate guarantees for the entire H\"older class including, in particular, both nonsmooth functions and those with Lipschitz continuous gradient. We also present the Universal Fast Gradient Method for SCO enjoying optimal efficiency estimates.