On Stochastic Variance Reduced Gradient Method for Semidefinite Optimization

TL;DR

This paper introduces a semi-stochastic variant of the SVRG method tailored for low-rank stochastic semidefinite optimization, providing theoretical guarantees and demonstrating superior practical performance over existing approaches.

Contribution

It develops a new SVRG variant with Option I for semidefinite optimization, establishing global linear convergence under certain conditions, and analyzes practical step size strategies.

Findings

The proposed SVRG method converges exponentially fast to a submanifold of the global minimum.

Numerical results show the method outperforms existing Option II and other algorithms.

The analysis includes effects of mini-batch size, update frequency, and step size strategies.

Abstract

The low-rank stochastic semidefinite optimization has attracted rising attention due to its wide range of applications. The nonconvex reformulation based on the low-rank factorization, significantly improves the computational efficiency but brings some new challenge to the analysis. The stochastic variance reduced gradient (SVRG) method has been regarded as one of the most effective methods. SVRG in general consists of two loops, where a reference full gradient is first evaluated in the outer loop and then used to yield a variance reduced estimate of the current gradient in the inner loop. Two options have been suggested to yield the output of the inner loop, where Option I sets the output as its last iterate, and Option II yields the output via random sampling from all the iterates in the inner loop. However, there is a significant gap between the theory and practice of SVRG when adapted to the stochastic semidefinite programming (SDP). SVRG practically works better with Option I, while most of existing theoretical results focus on Option II. In this paper, we fill this gap via exploiting a new semi-stochastic variant of the original SVRG with Option I adapted to the semidefinite optimization. Equipped with this, we establish the global linear submanifold convergence (i.e., converging exponentially fast to a submanifold of a global minimum under the orthogonal group action) of the proposed SVRG method, given a provable initialization scheme and under certain smoothness and restricted strongly convex assumptions. Our analysis includes the effects of the mini-batch size and update frequency in the inner loop as well as two practical step size strategies, the fixed and stabilized Barzilai-Borwein step sizes. Some numerical results in matrix sensing demonstrate the efficiency of proposed SVRG method outperforming Option II counterpart as well as others.