Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provably

TL;DR

This paper introduces BFGD, a first-order gradient descent method for optimizing convex functions over low-rank matrices via non-convex factorization, with provable convergence guarantees.

Contribution

It proposes the BFGD algorithm for low-rank matrix optimization and establishes its convergence properties under certain conditions, with practical initialization schemes.

Findings

BFGD achieves local sublinear convergence for smooth functions.

BFGD attains linear convergence when functions are also strongly convex.

Efficient initialization schemes enable practical application of the method.

Abstract

A rank-$r$ matrix $X \in \mathbb{R}^{m \times n}$ can be written as a product $U V^\top$, where $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{n \times r}$. One could exploit this observation in optimization: e.g., consider the minimization of a convex function $f(X)$ over rank-$r$ matrices, where the set of rank-$r$ matrices is modeled via the factorization $UV^\top$. Though such parameterization reduces the number of variables, and is more computationally efficient (of particular interest is the case $r \ll \min\{m, n\}$), it comes at a cost: $f(UV^\top)$ becomes a non-convex function w.r.t. $U$ and $V$. We study such parameterization for optimization of generic convex objectives $f$, and focus on first-order, gradient descent algorithmic solutions. We propose the Bi-Factored Gradient Descent (BFGD) algorithm, an efficient first-order method that operates on the $U, V$ factors. We show that when $f$ is (restricted) smooth, BFGD has local sublinear convergence, and linear convergence when $f$ is both (restricted) smooth and (restricted) strongly convex. For several key applications, we provide simple and efficient initialization schemes that provide approximate solutions good enough for the above convergence results to hold.