Dimension Reduction by Mutual Information Feature Extraction

TL;DR

This paper introduces MIFX, a mutual information-based feature extraction method that uses a gradient ascent approach with one-dimensional MI estimates, showing robust performance across various datasets.

Contribution

The paper proposes a novel component-by-component gradient ascent algorithm for MI-based feature extraction using 1D MI estimates, improving robustness and efficiency.

Findings

MIFX outperforms or matches existing methods on UCI datasets.

The method effectively balances dependency maximization and redundancy minimization.

Robust performance across diverse high-dimensional datasets.

Abstract

During the past decades, to study high-dimensional data in a large variety of problems, researchers have proposed many Feature Extraction algorithms. One of the most effective approaches for optimal feature extraction is based on mutual information (MI). However it is not always easy to get an accurate estimation for high dimensional MI. In terms of MI, the optimal feature extraction is creating a feature set from the data which jointly have the largest dependency on the target class and minimum redundancy. In this paper, a component-by-component gradient ascent method is proposed for feature extraction which is based on one-dimensional MI estimates. We will refer to this algorithm as Mutual Information Feature Extraction (MIFX). The performance of this proposed method is evaluated using UCI databases. The results indicate that MIFX provides a robust performance over different data sets which are almost always the best or comparable to the best ones.