An Improvement on the Hotelling $T^2$ Test Using the Ledoit-Wolf Nonlinear Shrinkage Estimator

TL;DR

This paper enhances Hotelling's $T^2$ test for high-dimensional data by integrating the Ledoit-Wolf nonlinear shrinkage estimator, resulting in improved performance over traditional methods across various covariance structures.

Contribution

The paper introduces a modified Hotelling's $T^2$ test utilizing the Ledoit-Wolf nonlinear shrinkage estimator, addressing eigenvalue inconsistency issues in high-dimensional settings.

Findings

The new test outperforms previous methods for diverse covariance matrices.

Empirical results show dominance for sub-Gaussian data.

Effective across matrices with different condition numbers and eigenvalue structures.

Abstract

Hotelling's $T^2$ test is a classical approach for discriminating the means of two multivariate normal samples that share a population covariance matrix. Hotelling's test is not ideal for high-dimensional samples because the eigenvalues of the estimated sample covariance matrix are inconsistent estimators for their population counterparts. We replace the sample covariance matrix with the nonlinear shrinkage estimator of Ledoit and Wolf 2020. We observe empirically for sub-Gaussian data that the resulting algorithm dominates past methods (Bai and Saranadasa 1996, Chen and Qin 2010, and Li et al. 2020) for a family of population covariance matrices that includes matrices with high or low condition number and many or few nontrivial -- i.e., spiked -- eigenvalues.