An Optimal Hybrid Variance-Reduced Algorithm for Stochastic Composite Nonconvex Optimization

TL;DR

This paper introduces a simplified hybrid variance-reduced proximal gradient algorithm for stochastic nonconvex optimization, reducing gradient evaluations and achieving optimal complexity bounds.

Contribution

It presents a new variant of the hybrid SARAH estimator that simplifies the algorithm and improves efficiency by requiring only two samples per iteration.

Findings

Achieves optimal stochastic oracle complexity bounds.

Reduces stochastic gradient evaluations per iteration.

Simplifies the hybrid variance-reduced algorithm.

Abstract

In this note we propose a new variant of the hybrid variance-reduced proximal gradient method in [7] to solve a common stochastic composite nonconvex optimization problem under standard assumptions. We simply replace the independent unbiased estimator in our hybrid- SARAH estimator introduced in [7] by the stochastic gradient evaluated at the same sample, leading to the identical momentum-SARAH estimator introduced in [2]. This allows us to save one stochastic gradient per iteration compared to [7], and only requires two samples per iteration. Our algorithm is very simple and achieves optimal stochastic oracle complexity bound in terms of stochastic gradient evaluations (up to a constant factor). Our analysis is essentially inspired by [7], but we do not use two different step-sizes.