Estimate Sequences for Stochastic Composite Optimization: Variance Reduction, Acceleration, and Robustness to Noise

TL;DR

This paper introduces a unified framework for stochastic convex composite optimization using estimate sequences, leading to new algorithms with improved convergence, acceleration, and robustness to noise.

Contribution

It extends Nesterov's estimate sequence concept to stochastic methods, providing a unified convergence proof and developing new accelerated, noise-robust algorithms.

Findings

Unified convergence analysis for stochastic methods

New accelerated algorithms with optimal complexity

Algorithms robust to stochastic noise

Abstract

In this paper, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization by extending the concept of estimate sequence introduced by Nesterov. More precisely, we interpret a large class of stochastic optimization methods as procedures that iteratively minimize a surrogate of the objective, which covers the stochastic gradient descent method and variants of the incremental approaches SAGA, SVRG, and MISO/Finito/SDCA. This point of view has several advantages: (i) we provide a simple generic proof of convergence for all of the aforementioned methods; (ii) we naturally obtain new algorithms with the same guarantees; (iii) we derive generic strategies to make these algorithms robust to stochastic noise, which is useful when data is corrupted by small random perturbations. Finally, we propose a new accelerated stochastic gradient descent algorithm and an accelerated SVRG algorithm with optimal complexity that is robust to stochastic noise.