Implicit regularization and solution uniqueness in over-parameterized matrix sensing

TL;DR

This paper investigates how the PSD constraint alone can ensure unique low-rank matrix recovery in over-parameterized matrix sensing, challenging the notion that implicit regularization from algorithmic choices is necessary.

Contribution

It demonstrates that under certain conditions, the PSD constraint guarantees solution uniqueness without relying on implicit regularization from the algorithm.

Findings

PSD constraint alone ensures unique rank-r solution

Over-parameterization does not hinder solution uniqueness under certain conditions

Implicit regularization may not be necessary for low-rank recovery

Abstract

We consider whether algorithmic choices in over-parameterized linear matrix factorization introduce implicit regularization. We focus on noiseless matrix sensing over rank-$r$ positive semi-definite (PSD) matrices in $\mathbb{R}^{n \times n}$, with a sensing mechanism that satisfies restricted isometry properties (RIP). The algorithm we study is \emph{factored gradient descent}, where we model the low-rankness and PSD constraints with the factorization $UU^\top$, for $U \in \mathbb{R}^{n \times r}$. Surprisingly, recent work argues that the choice of $r \leq n$ is not pivotal: even setting $U \in \mathbb{R}^{n \times n}$ is sufficient for factored gradient descent to find the rank-$r$ solution, which suggests that operating over the factors leads to an implicit regularization. In this contribution, we provide a different perspective to the problem of implicit regularization. We show that under certain conditions, the PSD constraint by itself is sufficient to lead to a unique rank-$r$ matrix recovery, without implicit or explicit low-rank regularization. \emph{I.e.}, under assumptions, the set of PSD matrices, that are consistent with the observed data, is a singleton, regardless of the algorithm used.