Sparse Centroid-Encoder: A Nonlinear Model for Feature Selection

TL;DR

The paper introduces Sparse Centroid-Encoder (SCE), a nonlinear feature selection method that effectively identifies discriminative features across diverse multi-modal datasets, outperforming existing techniques in classification accuracy.

Contribution

It proposes a novel sparse optimization approach within the Centroid-Encoder framework for feature selection, capable of handling multi-modal data and outperforming state-of-the-art methods.

Findings

SCE achieves higher classification accuracy than competing methods.

SCE effectively extracts informative features from complex multi-modal datasets.

The method demonstrates versatility across biological, image, speech, and sensor data.

Abstract

Autoencoders have been widely used as a nonlinear tool for data dimensionality reduction. While autoencoders don't utilize the label information, Centroid-Encoders (CE)\cite{ghosh2022supervised} use the class label in their learning process. In this study, we propose a sparse optimization using the Centroid-Encoder architecture to determine a minimal set of features that discriminate between two or more classes. The resulting algorithm, Sparse Centroid-Encoder (SCE), extracts discriminatory features in groups using a sparsity inducing $\ell_1$-norm while mapping a point to its class centroid. One key attribute of SCE is that it can extract informative features from a multi-modal data set, i.e., data sets whose classes appear to have multiple clusters. The algorithm is applied to a wide variety of real world data sets, including single-cell data, high dimensional biological data, image data, speech data, and accelerometer sensor data. We compared our method to various state-of-the-art feature selection techniques, including supervised Concrete Autoencoders (SCAE), Feature Selection Network (FsNet), deep feature selection (DFS), Stochastic Gate (STG), and LassoNet. We empirically showed that SCE features often produced better classification accuracy than other methods on sequester test set.