Sparsifying the Fisher Linear Discriminant by Rotation

TL;DR

This paper introduces a rotation-based method to induce sparsity in high-dimensional data, enhancing the performance of existing classifiers by leveraging principal components for improved classification accuracy.

Contribution

The paper proposes a novel rotate-and-solve approach using principal components to create sparsity, applicable to any high-dimensional classifier, with theoretical and empirical validation.

Findings

Rotations effectively induce sparsity in high-dimensional data.

The method improves classification accuracy over existing rules.

The approach is robust to different sparsity levels.

Abstract

Many high dimensional classification techniques have been proposed in the literature based on sparse linear discriminant analysis (LDA). To efficiently use them, sparsity of linear classifiers is a prerequisite. However, this might not be readily available in many applications, and rotations of data are required to create the needed sparsity. In this paper, we propose a family of rotations to create the required sparsity. The basic idea is to use the principal components of the sample covariance matrix of the pooled samples and its variants to rotate the data first and to then apply an existing high dimensional classifier. This rotate-and-solve procedure can be combined with any existing classifiers, and is robust against the sparsity level of the true model. We show that these rotations do create the sparsity needed for high dimensional classifications and provide theoretical understanding why such a rotation works empirically. The effectiveness of the proposed method is demonstrated by a number of simulated and real data examples, and the improvements of our method over some popular high dimensional classification rules are clearly shown.