Best Subset Selection: Optimal Pursuit for Feature Selection and Elimination

TL;DR

This paper proposes optimal criteria for feature selection and elimination in best subset selection, improving algorithm performance without extra computational cost across multiple statistical and machine learning tasks.

Contribution

It introduces new optimal criteria for feature entry and exit, enhancing best subset selection algorithms with theoretical guarantees and practical performance gains.

Findings

Enhanced algorithms preserve original theoretical properties.

Achieve significant performance improvements.

Applicable across diverse tasks like compressed sensing and sparse regression.

Abstract

This paper introduces two novel criteria: one for feature selection and another for feature elimination in the context of best subset selection, which is a benchmark problem in statistics and machine learning. From the perspective of optimization, we revisit the classical selection and elimination criteria in traditional best subset selection algorithms, revealing that these classical criteria capture only partial variations of the objective function after the entry or exit of features. By formulating and solving optimization subproblems for feature entry and exit exactly, new selection and elimination criteria are proposed, proved as the optimal decisions for the current entry-and-exit process compared to classical criteria. Replacing the classical selection and elimination criteria with the proposed ones generates a series of enhanced best subset selection algorithms. These generated algorithms not only preserve the theoretical properties of the original algorithms but also achieve significant meta-gains without increasing computational cost across various scenarios and evaluation metrics on multiple tasks such as compressed sensing and sparse regression.