Optimal Test-Data Piling in HDLSS Classification with Covariance Heterogeneity

TL;DR

This paper investigates data piling phenomena in high-dimensional classification with heterogeneous covariance, identifying an optimal direction for perfect separation and proposing an algorithm to find it using training data.

Contribution

It extends the understanding of data piling to heterogeneous covariance structures and introduces a method to compute the optimal piling direction in high-dimensional settings.

Findings

Optimal direction maximizes class separation in data piling.

Imbalance of tail eigenvalues is the main obstacle to finding the optimal direction.

Proposed algorithm effectively identifies the optimal direction in simulations.

Abstract

This work addresses a longstanding question in high-dimensional linear classification: Is perfect classification achievable in heterogeneous covariance structures? We focus on the phenomenon of data piling, where projected data points collapse onto discrete values. We provide a comprehensive characterization of two distinct types of data piling. The first type of data piling refers to the phenomenon where projecting the training data onto a certain direction yields exactly two distinct values-one for each class. This occurs universally when the data dimension $p$ exceeds the sample size $n$. The second type concerns independent test data and arises asymptotically as $p \to \infty$ with fixed $n$. While previous work established the existence of such double data piling under homogeneously spiked covariance structures using negatively ridged classifiers, our analysis extends to the more general and realistic case of heterogeneous covariance. We identify an optimal direction among all piling directions that maximizes the separation between test data piles, which is called the Second Maximal Data Piling direction. An algorithm based on data splitting is proposed to compute this direction using only training data. Our analysis reveals a key insight: the main obstacle to discovering this direction is the imbalance of the tail eigenvalues, rather than differences in spike count, spike magnitude, or the alignment of leading eigenspaces. Extensive simulations confirm our theoretical results and demonstrate the effectiveness of the proposed classifier across a wide range of high-dimensional scenarios.