Discovering Structure in High-Dimensional Data Through Correlation Explanation

TL;DR

This paper presents CorEx, an unsupervised, scalable method that uncovers hierarchical, meaningful structures in high-dimensional data by optimizing an information-theoretic objective based on multivariate mutual information.

Contribution

It introduces CorEx, a novel unsupervised approach that learns hierarchical representations without model assumptions, suitable for high-dimensional datasets.

Findings

Automatically discovers meaningful data structures.

Effective across diverse data sources like personality tests, DNA, and language.

Scales linearly with data variables.

Abstract

We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective. Intuitively, the optimization searches for a set of latent factors that best explain the correlations in the data as measured by multivariate mutual information. The method is unsupervised, requires no model assumptions, and scales linearly with the number of variables which makes it an attractive approach for very high dimensional systems. We demonstrate that Correlation Explanation (CorEx) automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language.