On the analysis of optimization with fixed-rank matrices: a quotient geometric view

TL;DR

This paper introduces a Riemannian gradient descent algorithm for fixed-rank matrix optimization, analyzing its convergence and efficiency through a quotient geometric perspective, and demonstrating superior performance over Euclidean methods.

Contribution

The paper presents a novel Riemannian gradient descent approach based on quotient geometry for fixed-rank matrices, with explicit update rules and convergence analysis.

Findings

RGD algorithm is faster than Euclidean gradient descent.

RGD does not require balancing techniques for efficiency.

Guaranteed linear convergence for matrix sensing and completion.

Abstract

We study a type of Riemannian gradient descent (RGD) algorithm, designed through Riemannian preconditioning, for optimization on $\mathcal{M}_k^{m\times n}$ -- the set of $m\times n$ real matrices with a fixed rank $k$. Our analysis is based on a quotient geometric view of $\mathcal{M}_k^{m\times n}$: by identifying this set with the quotient manifold of a two-term product space $\mathbb{R}_*^{m\times k}\times \mathbb{R}_*^{n\times k}$ of matrices with full column rank via matrix factorization, we find an explicit form for the update rule of the RGD algorithm, which leads to a novel approach to analysing their convergence behavior in rank-constrained optimization. We then deduce some interesting properties that reflect how RGD distinguishes from other matrix factorization algorithms such as those based on the Euclidean geometry. In particular, we show that the RGD algorithm is not only faster than Euclidean gradient descent but also does not rely on balancing techniques to ensure its efficiency while the latter does. We further show that this RGD algorithm is guaranteed to solve matrix sensing and matrix completion problems with linear convergence rate under the restricted positive definiteness property. Numerical experiments on matrix sensing and completion are provided to demonstrate these properties.