Implicit Regularization in Matrix Sensing via Mirror Descent

TL;DR

This paper analyzes how mirror descent implicitly regularizes matrix sensing problems, showing it converges to low-rank solutions similar to nuclear norm minimization, with explicit characterizations of the bias in certain cases.

Contribution

It provides a potential-based analysis of mirror descent in matrix sensing, revealing its implicit bias towards low-rank solutions and connecting it to nuclear norm minimization.

Findings

Mirror descent converges to solutions minimizing a combination of nuclear norm, Frobenius norm, and von Neumann entropy.

Under certain conditions, mirror descent recovers low-rank matrices comparable to nuclear norm minimization.

Gradient descent with factorized parametrization approximates mirror descent, allowing explicit bias characterization.

Abstract

We study discrete-time mirror descent applied to the unregularized empirical risk in matrix sensing. In both the general case of rectangular matrices and the particular case of positive semidefinite matrices, a simple potential-based analysis in terms of the Bregman divergence allows us to establish convergence of mirror descent -- with different choices of the mirror maps -- to a matrix that, among all global minimizers of the empirical risk, minimizes a quantity explicitly related to the nuclear norm, the Frobenius norm, and the von Neumann entropy. In both cases, this characterization implies that mirror descent, a first-order algorithm minimizing the unregularized empirical risk, recovers low-rank matrices under the same set of assumptions that are sufficient to guarantee recovery for nuclear-norm minimization. When the sensing matrices are symmetric and commute, we show that gradient descent with full-rank factorized parametrization is a first-order approximation to mirror descent, in which case we obtain an explicit characterization of the implicit bias of gradient flow as a by-product.