Top-$k$ Regularization for Supervised Feature Selection

TL;DR

This paper proposes a novel top-$k$ regularization method for supervised feature selection that enhances the selection of informative features and models complex nonlinear relationships, with proven theoretical bounds and extensive empirical validation.

Contribution

Introduces a new top-$k$ regularization approach that improves feature selection by balancing representativeness and inter-feature correlations, with theoretical guarantees and practical effectiveness.

Findings

Effective in selecting informative features across datasets

Provides a uniform approximation error bound for high-dimensional functions

Demonstrates stability and superiority over existing methods

Abstract

Feature selection identifies subsets of informative features and reduces dimensions in the original feature space, helping provide insights into data generation or a variety of domain problems. Existing methods mainly depend on feature scoring functions or sparse regularizations; nonetheless, they have limited ability to reconcile the representativeness and inter-correlations of features. In this paper, we introduce a novel, simple yet effective regularization approach, named top-$k$ regularization, to supervised feature selection in regression and classification tasks. Structurally, the top-$k$ regularization induces a sub-architecture on the architecture of a learning model to boost its ability to select the most informative features and model complex nonlinear relationships simultaneously. Theoretically, we derive and mathematically prove a uniform approximation error bound for using this approach to approximate high-dimensional sparse functions. Extensive experiments on a wide variety of benchmarking datasets show that the top-$k$ regularization is effective and stable for supervised feature selection.