Estimate Sequences for Variance-Reduced Stochastic Composite Optimization

TL;DR

This paper introduces a unified framework for analyzing and developing stochastic gradient algorithms for convex composite optimization, extending Nesterov's estimate sequences to improve convergence proofs, adaptivity, robustness, and acceleration.

Contribution

It extends the concept of estimate sequences to stochastic composite optimization, providing a unified analysis, new algorithms, and strategies for robustness and acceleration.

Findings

Provided a generic convergence proof for stochastic gradient methods.

Developed an adaptive SVRG variant for strong convexity.

Derived new accelerated algorithms based on the estimate sequence framework.

Abstract

In this paper, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization by extending the concept of estimate sequence introduced by Nesterov. This point of view covers the stochastic gradient descent method, variants of the approaches SAGA, SVRG, and has several advantages: (i) we provide a generic proof of convergence for the aforementioned methods; (ii) we show that this SVRG variant is adaptive to strong convexity; (iii) we naturally obtain new algorithms with the same guarantees; (iv) we derive generic strategies to make these algorithms robust to stochastic noise, which is useful when data is corrupted by small random perturbations. Finally, we show that this viewpoint is useful to obtain new accelerated algorithms in the sense of Nesterov.