SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient

TL;DR

SARAH introduces a recursive stochastic gradient method that achieves linear convergence for finite-sum minimization problems without needing to store past gradients, showing improved efficiency over existing methods.

Contribution

The paper presents SARAH, a new recursive stochastic gradient algorithm with proven linear convergence and a practical variant SARAH+ that outperforms existing stochastic methods.

Findings

SARAH achieves linear convergence under strong convexity.

SARAH+ demonstrates practical efficiency in experiments.

SARAH does not require storing past gradients unlike SAG/SAGA.

Abstract

In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems. Different from the vanilla SGD and other modern stochastic methods such as SVRG, S2GD, SAG and SAGA, SARAH admits a simple recursive framework for updating stochastic gradient estimates; when comparing to SAG/SAGA, SARAH does not require a storage of past gradients. The linear convergence rate of SARAH is proven under strong convexity assumption. We also prove a linear convergence rate (in the strongly convex case) for an inner loop of SARAH, the property that SVRG does not possess. Numerical experiments demonstrate the efficiency of our algorithm.