Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems

TL;DR

This paper proves that a modified stochastic gradient descent algorithm for low-rank matrix problems converges globally from random initialization under broad sampling conditions, with practical experiments demonstrating its efficiency.

Contribution

The paper introduces a step size scheme for SGD on low-rank least-squares problems and proves its global convergence from random start under broad sampling conditions.

Findings

Convergence within $O(rac{1}{\u03b5} n \u2206 )$ steps with high probability.

Relation of the modified SGD to stochastic power iteration.

Experimental validation of runtime and convergence.

Abstract

Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation. In this paper, we exhibit a step size scheme for SGD on a low-rank least-squares problem, and we prove that, under broad sampling conditions, our method converges globally from a random starting point within $O(\epsilon^{-1} n \log n)$ steps with constant probability for constant-rank problems. Our modification of SGD relates it to stochastic power iteration. We also show experiments to illustrate the runtime and convergence of the algorithm.