Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing

TL;DR

This paper demonstrates that for asymmetric low-rank matrix sensing, gradient descent with spectral initialization converges linearly without explicit regularization, as the factors naturally stay balanced during optimization, simplifying the approach.

Contribution

The paper provides a theoretical justification that balancing regularization is unnecessary in asymmetric matrix sensing when using gradient descent with spectral initialization.

Findings

Gradient descent converges linearly without balancing regularization.

Factors remain balanced automatically during the optimization process.

Analysis based on a new distance metric accounting for invertible transform ambiguity.

Abstract

Low-rank matrix estimation plays a central role in various applications across science and engineering. Recently, nonconvex formulations based on matrix factorization are provably solved by simple gradient descent algorithms with strong computational and statistical guarantees. However, when the low-rank matrices are asymmetric, existing approaches rely on adding a regularization term to balance the scale of the two matrix factors which in practice can be removed safely without hurting the performance when initialized via the spectral method. In this paper, we provide a theoretical justification to this for the matrix sensing problem, which aims to recover a low-rank matrix from a small number of linear measurements. As long as the measurement ensemble satisfies the restricted isometry property, gradient descent -- in conjunction with spectral initialization -- converges linearly without the need of explicitly promoting balancedness of the factors; in fact, the factors stay balanced automatically throughout the execution of the algorithm. Our analysis is based on analyzing the evolution of a new distance metric that directly accounts for the ambiguity due to invertible transforms, and might be of independent interest.