Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbations

TL;DR

This paper demonstrates that implicit regularization in overparameterized models, specifically in matrix sensing, makes solutions robust to perturbations and noise, improving understanding of generalization and efficiency of gradient-based methods.

Contribution

It introduces a perturbed gradient flow framework that captures noise effects, providing sharper complexity bounds and robustness analysis for matrix sensing with small initializations.

Findings

Perturbed gradient flow simplifies analysis and improves sample complexity.

Factorized gradient descent is robust to measurement noise and perturbations.

Mini-batch stochastic gradient descent further reduces sample complexity.

Abstract

Several key questions remain unanswered regarding overparameterized learning models. It is unclear how (stochastic) gradient descent finds solutions that generalize well, and in particular the role of small random initializations. Matrix sensing, which is the problem of reconstructing a low-rank matrix from a few linear measurements, has become a standard prototypical setting to study these phenomena. Previous works have shown that matrix sensing can be solved by factorized gradient descent, provided the random initialization is extremely small. In this paper, we find that factorized gradient descent is highly robust to certain perturbations. This lets us use a perturbation term to capture both the effects of imperfect measurements, discretization by gradient descent, and other noise, resulting in a general formulation which we call \textit{perturbed gradient flow}. We find that not only is this equivalent formulation easier to work with, but it leads to sharper sample and time complexities than previous work, handles moderately small initializations, and the results are naturally robust to perturbations such as noisy measurements or changing measurement matrices. Finally, we also analyze mini-batch stochastic gradient descent using the formulation, where we find improved sample complexity.