Permutation-based multi-objective evolutionary feature selection for high-dimensional data

TL;DR

This paper introduces a novel multi-objective evolutionary feature selection method based on permutation importance, effectively capturing feature interactions to improve model performance and efficiency in high-dimensional data analysis.

Contribution

It extends permutation feature importance to evaluate feature subsets and employs a multi-objective evolutionary algorithm for optimal feature selection.

Findings

Outperforms 9 established feature selection methods on 24 datasets.

Balances accuracy and computational efficiency effectively.

Captures feature interactions better than traditional methods.

Abstract

Feature selection is a critical step in the analysis of high-dimensional data, where the number of features often vastly exceeds the number of samples. Effective feature selection not only improves model performance and interpretability but also reduces computational costs and mitigates the risk of overfitting. In this context, we propose a novel feature selection method for high-dimensional data, based on the well-known permutation feature importance approach, but extending it to evaluate subsets of attributes rather than individual features. This extension more effectively captures how interactions among features influence model performance. The proposed method employs a multi-objective evolutionary algorithm to search for candidate feature subsets, with the objectives of maximizing the degradation in model performance when the selected features are shuffled, and minimizing the cardinality of the feature subset. The effectiveness of our method has been validated on a set of 24 publicly available high-dimensional datasets for classification and regression tasks, and compared against 9 well-established feature selection methods designed for high-dimensional problems, including the conventional permutation feature importance method. The results demonstrate the ability of our approach in balancing accuracy and computational efficiency, providing a powerful tool for feature selection in complex, high-dimensional datasets.