Spectrally Robust Covariance Shrinkage for Hotelling's $T^2$ in High Dimensions

TL;DR

This paper develops a new covariance shrinkage method for Hotelling's $T^2$ test in high-dimensional settings, achieving significant power improvements over existing methods through a variational approach and random matrix theory.

Contribution

It introduces a practical finite-sample shrinker that asymptotically maximizes test power and saturates theoretical bounds in high-dimensional regimes without restrictive assumptions.

Findings

Up to 50% power gain over competitors in simulations.

Method performs well on real-world data, including RSS dataset.

Theoretical analysis confirms asymptotic optimality of the shrinker.

Abstract

We investigate covariance shrinkage for Hotelling's $T^2$ in the regime where the data dimension $p$ and the sample size $n$ grow in a fixed ratio -- without assuming that the population covariance matrix is spiked or well-conditioned. When $p/n\to\phi \in (0,1)$, we propose a practical finite-sample shrinker that, for any maximum-entropy signal prior and any fixed significance level, (a) asymptotically maximizes power under Gaussian data, and (b) asymptotically saturates the Hanson--Wright lower bound on power in the more general sub-Gaussian case. Our approach is to formulate and solve a variational problem characterizing the optimal limiting shrinker, and to show that our finite-sample method consistently approximates this limit by extending recent local random matrix laws. Empirical studies on simulated and real-world data, including the Crawdad UMich/RSS data set, demonstrate up to a $50\%$ gain in power over leading linear and nonlinear competitors at a significance level of $10^{-4}$.