Accelerated Convex Optimization with Stochastic Gradients: Generalizing the Strong-Growth Condition

TL;DR

This paper introduces a new sufficient condition ensuring stochastic gradients do not hinder the convergence of Nesterov's accelerated method, broadening its applicability to constrained problems and finite-sum oracles.

Contribution

It generalizes the strong-growth condition, enabling the design of stochastic oracles for constrained and finite-sum problems without altering the core accelerated algorithm.

Findings

The new condition encompasses the strong-growth condition as a special case.

It allows modeling of constrained problems within accelerated stochastic optimization.

Facilitates the design of specialized stochastic oracles like SAGA.

Abstract

This paper presents a sufficient condition for stochastic gradients not to slow down the convergence of Nesterov's accelerated gradient method. The new condition has the strong-growth condition by Schmidt \& Roux as a special case, and it also allows us to (i) model problems with constraints and (ii) design new types of oracles (e.g., oracles for finite-sum problems such as SAGA). Our results are obtained by revisiting Nesterov's accelerated algorithm and are useful for designing stochastic oracles without changing the underlying first-order method.