An Information-theoretic Approach to Unsupervised Feature Selection for High-Dimensional Data

TL;DR

This paper introduces an information-theoretic method for unsupervised feature selection in high-dimensional data, leveraging common information measures and neural network training to identify informative features.

Contribution

It proposes a novel approach combining information theory and neural networks to extract hidden shared structures in high-dimensional data for feature selection.

Findings

Effective in identifying shared structures in high-dimensional data

Connections established with PCA, HGR correlation, and functional maps

Validated through numerical simulations

Abstract

In this paper, we propose an information-theoretic approach to design the functional representations to extract the hidden common structure shared by a set of random variables. The main idea is to measure the common information between the random variables by Watanabe's total correlation, and then find the hidden attributes of these random variables such that the common information is reduced the most given these attributes. We show that these attributes can be characterized by an exponential family specified by the eigen-decomposition of some pairwise joint distribution matrix. Then, we adopt the log-likelihood functions for estimating these attributes as the desired functional representations of the random variables, and show that such representations are informative to describe the common structure. Moreover, we design both the multivariate alternating conditional expectation (MACE) algorithm to compute the proposed functional representations for discrete data, and a novel neural network training approach for continuous or high-dimensional data. Furthermore, we show that our approach has deep connections to existing techniques, such as Hirschfeld-Gebelein-R\'{e}nyi (HGR) maximal correlation, linear principal component analysis (PCA), and consistent functional map, which establishes insightful connections between information theory and machine learning. Finally, the performances of our algorithms are validated by numerical simulations.