Finding Global Optima in Nonconvex Stochastic Semidefinite Optimization with Variance Reduction

TL;DR

This paper demonstrates that a variance-reduced stochastic gradient descent method can efficiently find global optima in nonconvex low-rank reformulations of semidefinite problems, with proven linear convergence.

Contribution

It introduces a variance reduction technique for stochastic gradient descent that guarantees global linear convergence in nonconvex semidefinite optimization.

Findings

Proves global linear convergence under certain conditions.

Shows effectiveness through experiments on simulations and real data.

Addresses scalability for large-scale semidefinite problems.

Abstract

There is a recent surge of interest in nonconvex reformulations via low-rank factorization for stochastic convex semidefinite optimization problem in the purpose of efficiency and scalability. Compared with the original convex formulations, the nonconvex ones typically involve much fewer variables, allowing them to scale to scenarios with millions of variables. However, it opens a new challenge that under what conditions the nonconvex stochastic algorithms may find the global optima effectively despite their empirical success in applications. In this paper, we provide an answer that a stochastic gradient descent method with variance reduction, can be adapted to solve the nonconvex reformulation of the original convex problem, with a \textit{global linear convergence}, i.e., converging to a global optimum exponentially fast, at a proper initial choice in the restricted strongly convex case. Experimental studies on both simulation and real-world applications on ordinal embedding are provided to show the effectiveness of the proposed algorithms.