Beam Search for Feature Selection

TL;DR

This paper introduces a beam search method for feature selection that outperforms traditional forward selection, especially with correlated features, leading to more efficient models with fewer features.

Contribution

It proposes a novel beam search approach for feature selection and demonstrates its effectiveness over forward selection through theoretical and empirical analysis.

Findings

Beam search outperforms forward selection in correlated feature scenarios.

Models with fewer features selected by beam search achieve comparable accuracy.

Significant reduction in feature set size without loss of performance.

Abstract

In this paper, we present and prove some consistency results about the performance of classification models using a subset of features. In addition, we propose to use beam search to perform feature selection, which can be viewed as a generalization of forward selection. We apply beam search to both simulated and real-world data, by evaluating and comparing the performance of different classification models using different sets of features. The results demonstrate that beam search could outperform forward selection, especially when the features are correlated so that they have more discriminative power when considered jointly than individually. Moreover, in some cases classification models could obtain comparable performance using only ten features selected by beam search instead of hundreds of original features.