New Hard-thresholding Rules based on Data Splitting in High-dimensional Imbalanced Classification

TL;DR

This paper addresses the challenge of imbalanced high-dimensional classification by proposing a new data splitting based hard-thresholding rule that improves classification accuracy and feature selection efficiency.

Contribution

It introduces a novel data splitting technique for hard-thresholding in imbalanced high-dimensional classification, demonstrating asymptotic optimality and improved finite-sample performance.

Findings

The proposed method reduces misclassification rates in imbalanced high-dimensional data.

It outperforms existing methods or matches their performance with fewer features.

The method is computationally efficient and effective in real data applications.

Abstract

In binary classification, imbalance refers to situations in which one class is heavily under-represented. This issue is due to either a data collection process or because one class is indeed rare in a population. Imbalanced classification frequently arises in applications such as biology, medicine, engineering, and social sciences. In this paper, for the first time, we theoretically study the impact of imbalance class sizes on the linear discriminant analysis (LDA) in high dimensions. We show that due to data scarcity in one class, referred to as the minority class, and high-dimensionality of the feature space, the LDA ignores the minority class yielding a maximum misclassification rate. We then propose a new construction of hard-thresholding rules based on a data splitting technique that reduces the large difference between the misclassification rates. We show that the proposed method is asymptotically optimal. We further study two well-known sparse versions of the LDA in imbalanced cases. We evaluate the finite-sample performance of different methods using simulations and by analyzing two real data sets. The results show that our method either outperforms its competitors or has comparable performance based on a much smaller subset of selected features, while being computationally more efficient.