Fast Incremental Method for Nonconvex Optimization

TL;DR

This paper introduces a novel analysis of the SAGA algorithm, demonstrating faster convergence to stationary points in nonconvex optimization and linear convergence for certain problem classes, including practical variants.

Contribution

First analysis showing fast convergence of an incremental gradient method for nonconvex problems, including global convergence in special cases and practical algorithm variants.

Findings

SAGA converges faster than gradient descent and stochastic gradient descent.

Linear convergence to the global optimum for Polyak's class of nonconvex problems.

Analysis includes regularized and minibatch variants of SAGA.

Abstract

We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form $\min_x \sum_i f_i(x)$. Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and stochastic gradient descent. We also discuss a Polyak's special class of nonconvex problems for which SAGA converges at a linear rate to the global optimum. Finally, we analyze the practically valuable regularized and minibatch variants of SAGA. To our knowledge, this paper presents the first analysis of fast convergence for an incremental aggregated gradient method for nonconvex problems.