Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

TL;DR

This paper introduces accelerated stochastic algorithms, RapGrad and RapDual, that significantly reduce gradient and projection computations for nonconvex finite-sum and multi-block problems, especially when the condition number is large.

Contribution

The paper proposes novel randomized accelerated methods, RapGrad and RapDual, with improved complexity bounds for nonconvex optimization, outperforming existing deterministic and randomized algorithms.

Findings

RapGrad reduces gradient computations by up to O(√m) compared to deterministic methods.

RapDual decreases projection subproblems by up to O(√m) over deterministic counterparts.

Complexity bounds are new and demonstrate efficiency gains in large-scale nonconvex problems.

Abstract

In this paper, we present new stochastic methods for solving two important classes of nonconvex optimization problems. We first introduce a randomized accelerated proximal gradient (RapGrad) method for solving a class of nonconvex optimization problems consisting of the sum of $m$ component functions, and show that it can significantly reduce the number of gradient computations especially when the condition number $L/\mu$ (i.e., the ratio between the Lipschitz constant and negative curvature) is large. More specifically, RapGrad can save up to ${\cal O}(\sqrt{m})$ gradient computations than existing deterministic nonconvex accelerated gradient methods. Moreover, the number of gradient computations required by RapGrad can be ${\cal O}(m^\frac{1}{6} L^\frac{1}{2} / \mu^\frac{1}{2})$ (at least ${\cal O}(m^\frac{2}{3})$) times smaller than the best-known randomized nonconvex gradient methods when $L/\mu \ge m$. Inspired by RapGrad, we also develop a new randomized accelerated proximal dual (RapDual) method for solving a class of multi-block nonconvex optimization problems coupled with linear constraints. We demonstrate that RapDual can also save up to a factor of ${\cal O}(\sqrt{m})$ projection subproblems than its deterministic counterpart, where $m$ denotes the number of blocks. To the best of our knowledge, all these complexity results associated with RapGrad and RapDual seem to be new in the literature. We also illustrate potential advantages of these algorithms through our preliminary numerical experiments.