Optimal Covariance Estimation for Condition Number Loss in the Spiked Model

TL;DR

This paper develops an asymptotically optimal nonlinear shrinker for covariance matrix estimation under condition number loss in the spiked model, with applications to multi-user covariance estimation and linear discriminant analysis.

Contribution

It introduces a new nonlinear shrinker tailored for the spiked covariance model that is asymptotically optimal under the condition number loss, improving estimation accuracy.

Findings

Optimal shrinker depends on data aspect ratio and top eigenvalue.

Large aspect ratio leads to substantial eigenvalue shrinkage.

Diagonal covariance matrices can be optimal even with large eigenvalues.

Abstract

We study estimation of the covariance matrix under relative condition number loss $\kappa(\Sigma^{-1/2} \hat{\Sigma} \Sigma^{-1/2})$, where $\kappa(\Delta)$ is the condition number of matrix $\Delta$, and $\hat{\Sigma}$ and $\Sigma$ are the estimated and theoretical covariance matrices. Optimality in $\kappa$-loss provides optimal guarantees in two stylized applications: Multi-User Covariance Estimation and Multi-Task Linear Discriminant Analysis. We assume the so-called spiked covariance model for $\Sigma$, and exploit recent advances in understanding that model, to derive a nonlinear shrinker which is asymptotically optimal among orthogonally-equivariant procedures. In our asymptotic study, the number of variables $p$ is comparable to the number of observations $n$. The form of the optimal nonlinearity depends on the aspect ratio $\gamma=p/n$ of the data matrix and on the top eigenvalue of $\Sigma$. For $\gamma > 0.618...$, even dependence on the top eigenvalue can be avoided. The optimal shrinker has two notable properties. First, when $p/n \rightarrow \gamma \gg 1$ is large, it shrinks even very large eigenvalues substantially, by a factor $1/(1+\gamma)$. Second, even for moderate $\gamma$, certain highly statistically significant eigencomponents will be completely suppressed. We show that when $\gamma \gg 1$ is large, purely diagonal covariance matrices can be optimal, despite the top eigenvalues being large and the empirical eigenvalues being highly statistically significant. This aligns with practitioner experience. We identify intuitively reasonable procedures with small worst-case relative regret - the simplest being generalized soft thresholding having threshold at the bulk edge and slope $(1+\gamma)^{-1}$ above the bulk. For $\gamma < 2$ it has at most a few percent relative regret.