Non-convex Finite-Sum Optimization Via SCSG Methods

TL;DR

This paper introduces variants of the SCSG algorithm for non-convex finite-sum optimization, achieving improved complexity bounds and outperforming traditional stochastic gradient methods, especially at low target accuracies.

Contribution

The paper develops new SCSG variants with better complexity bounds for non-convex optimization and demonstrates their empirical superiority over existing methods.

Findings

SCSG variants outperform stochastic gradient descent in reaching stationary points.

The algorithms achieve lower complexity bounds than previous methods.

Empirical results show faster training and validation loss reduction on neural networks.

Abstract

We develop a class of algorithms, as variants of the stochastically controlled stochastic gradient (SCSG) methods (Lei and Jordan, 2016), for the smooth non-convex finite-sum optimization problem. Assuming the smoothness of each component, the complexity of SCSG to reach a stationary point with $\mathbb{E} \|\nabla f(x)\|^{2}\le \epsilon$ is $O\left (\min\{\epsilon^{-5/3}, \epsilon^{-1}n^{2/3}\}\right)$, which strictly outperforms the stochastic gradient descent. Moreover, SCSG is never worse than the state-of-the-art methods based on variance reduction and it significantly outperforms them when the target accuracy is low. A similar acceleration is also achieved when the functions satisfy the Polyak-Lojasiewicz condition. Empirical experiments demonstrate that SCSG outperforms stochastic gradient methods on training multi-layers neural networks in terms of both training and validation loss.