Optimal Shrinkage of Singular Values

TL;DR

This paper develops a theoretical framework for optimal singular value shrinkage in low-rank matrix recovery from noisy data, providing explicit formulas and demonstrating improved performance over traditional thresholding methods.

Contribution

It introduces a general method to evaluate and derive optimal nonlinear shrinkage functions for various loss functions in high-dimensional matrix denoising.

Findings

Optimal shrinkage formulas for Frobenius, nuclear, and operator norm losses.

Optimal nonlinearity shrinks singular values to rom the observed data.

Improved asymptotic mean square error compared to hard and soft thresholding.

Abstract

We consider recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE - square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level $\sigma$, the optimal nonlinearity for MSE loss simply shrinks each data singular value $y$ to $\sqrt{y^2-4n\sigma^2 }$ (or to 0 if $y<2\sqrt{n}\sigma$). This optimal nonlinearity guarantees an asymptotic MSE of $2nr\sigma^2$, which compares favorably with optimally tuned hard thresholding and optimally tuned soft thresholding, providing guarantees of $3nr\sigma^2$ and $6nr\sigma^2$, respectively. Our general method also allows one to evaluate optimal shrinkers numerically to arbitrary precision. As an example, we compute optimal shrinkers for the Schatten-p norm loss, for any p>0.