Significance Analysis of High-Dimensional, Low-Sample Size Partially Labeled Data

TL;DR

This paper introduces a significance testing method for high-dimensional, low-sample size data with partial labels, improving the detection of class differences by utilizing all available data more effectively.

Contribution

It proposes a novel significance analysis approach for partially labeled high-dimensional data, enhancing power while maintaining size, with theoretical and empirical validation.

Findings

The method outperforms traditional tests ignoring label information in simulations.

Theoretical analysis confirms the method's validity in high-dimensional, low-sample size settings.

Real data example demonstrates practical usefulness of the approach.

Abstract

Classification and clustering are both important topics in statistical learning. A natural question herein is whether predefined classes are really different from one another, or whether clusters are really there. Specifically, we may be interested in knowing whether the two classes defined by some class labels (when they are provided), or the two clusters tagged by a clustering algorithm (where class labels are not provided), are from the same underlying distribution. Although both are challenging questions for the high-dimensional, low-sample size data, there has been some recent development for both. However, when it is costly to manually place labels on observations, it is often that only a small portion of the class labels is available. In this article, we propose a significance analysis approach for such type of data, namely partially labeled data. Our method makes use of the whole data and tries to test the class difference as if all the labels were observed. Compared to a testing method that ignores the label information, our method provides a greater power, meanwhile, maintaining the size, illustrated by a comprehensive simulation study. Theoretical properties of the proposed method are studied with emphasis on the high-dimensional, low-sample size setting. Our simulated examples help to understand when and how the information extracted from the labeled data can be effective. A real data example further illustrates the usefulness of the proposed method.