Random-reshuffled SARAH does not need a full gradient computations

TL;DR

This paper introduces a variant of the SARAH algorithm that eliminates the need for full gradient computations by using randomized reshuffling and gradient aggregation, supported by theoretical analysis and numerical experiments.

Contribution

It proposes a new SARAH variant that removes full gradient computations through randomized reshuffling and gradient aggregation, with theoretical and experimental validation.

Findings

Reduces computational cost by avoiding full gradient calculations.

Maintains convergence properties with the new reshuffling strategy.

Demonstrates improved efficiency in numerical experiments.

Abstract

The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time. In this paper, we remove the necessity of a full gradient computation. This is achieved by using a randomized reshuffling strategy and aggregating stochastic gradients obtained in each epoch. The aggregated stochastic gradients serve as an estimate of a full gradient in the SARAH algorithm. We provide a theoretical analysis of the proposed approach and conclude the paper with numerical experiments that demonstrate the efficiency of this approach.