Block stochastic gradient iteration for convex and nonconvex optimization

TL;DR

This paper introduces a block stochastic gradient (BSG) method that combines stochastic gradient and block coordinate descent techniques to efficiently solve large-scale convex and nonconvex optimization problems, with proven convergence properties.

Contribution

The paper proposes a novel BSG method that integrates SG and BCD approaches, providing convergence guarantees for both convex and nonconvex problems, and demonstrating its effectiveness through numerical experiments.

Findings

Achieves the same convergence rate as SG for convex problems

Establishes convergence in expected violation of optimality for nonconvex problems

Shows effectiveness on problems like stochastic least squares, logistic regression, low-rank tensor recovery

Abstract

The stochastic gradient (SG) method can minimize an objective function composed of a large number of differentiable functions, or solve a stochastic optimization problem, to a moderate accuracy. The block coordinate descent/update (BCD) method, on the other hand, handles problems with multiple blocks of variables by updating them one at a time; when the blocks of variables are easier to update individually than together, BCD has a lower per-iteration cost. This paper introduces a method that combines the features of SG and BCD for problems with many components in the objective and with multiple (blocks of) variables. Specifically, a block stochastic gradient (BSG) method is proposed for solving both convex and nonconvex programs. At each iteration, BSG approximates the gradient of the differentiable part of the objective by randomly sampling a small set of data or sampling a few functions from the sum term in the objective, and then, using those samples, it updates all the blocks of variables in either a deterministic or a randomly shuffled order. Its convergence for both convex and nonconvex cases are established in different senses. In the convex case, the proposed method has the same order of convergence rate as the SG method. In the nonconvex case, its convergence is established in terms of the expected violation of a first-order optimality condition. The proposed method was numerically tested on problems including stochastic least squares and logistic regression, which are convex, as well as low-rank tensor recovery and bilinear logistic regression, which are nonconvex.