Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

TL;DR

This paper establishes a theoretical framework demonstrating that projected gradient descent can efficiently solve low-rank matrix problems under broad conditions, with guarantees even from poor initializations.

Contribution

It provides general conditions under which projected gradient descent converges rapidly to useful solutions in nonconvex low-rank matrix problems, applicable to various models.

Findings

Convergence guarantees for projected gradient descent with suitable initialization.

Applicability to matrix regression, PCA, completion, decomposition, and clustering.

Simulation results confirm theoretical predictions.

Abstract

Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factorized optimization problem. The goal of this problem is to provide a general theoretical framework for understanding when such methods work well, and to characterize the nature of the resulting fixed point. We provide a simple set of conditions under which projected gradient descent, when given a suitable initialization, converges geometrically to a statistically useful solution. Our results are applicable even when the initial solution is outside any region of local convexity, and even when the problem is globally concave. Working in a non-asymptotic framework, we show that our conditions are satisfied for a wide range of concrete models, including matrix regression, structured PCA, matrix completion with real and quantized observations, matrix decomposition, and graph clustering problems. Simulation results show excellent agreement with the theoretical predictions.