Asymptotic Escape of Spurious Critical Points on the Low-rank Matrix Manifold

TL;DR

This paper demonstrates that Riemannian gradient descent on the low-rank matrix manifold almost surely avoids certain spurious critical points, and introduces methods to transform these points into strict saddles, supported by numerical evidence.

Contribution

It is the first work to show that vanilla Riemannian gradient descent can almost surely escape specific spurious critical points on the low-rank matrix manifold without modifications.

Findings

Gradient descent almost surely escapes spurious critical points.

Transforming spurious points into strict saddles facilitates escape.

Numerical experiments validate theoretical results.

Abstract

We show that on the manifold of fixed-rank and symmetric positive semi-definite matrices, the Riemannian gradient descent algorithm almost surely escapes some spurious critical points on the boundary of the manifold. Our result is the first to partially overcome the incompleteness of the low-rank matrix manifold without changing the vanilla Riemannian gradient descent algorithm. The spurious critical points are some rank-deficient matrices that capture only part of the eigen components of the ground truth. Unlike classical strict saddle points, they exhibit very singular behavior. We show that using the dynamical low-rank approximation and a rescaled gradient flow, some of the spurious critical points can be converted to classical strict saddle points in the parameterized domain, which leads to the desired result. Numerical experiments are provided to support our theoretical findings.