Information Theory in Density Destructors

TL;DR

This paper introduces density destructors, invertible transforms that simplify complex data distributions, and demonstrates their connection to information theory, enabling more accurate estimation of information measures and potential new optimization strategies.

Contribution

The paper presents a novel class of density destructors that simplify data distributions and links them to classical information theory, improving estimation of information quantities.

Findings

Density destructors effectively simplify complex data distributions.

They enable more accurate estimation of mutual information and total correlation.

Experiments show superiority over existing methods in information measure estimation.

Abstract

Density destructors are differentiable and invertible transforms that map multivariate PDFs of arbitrary structure (low entropy) into non-structured PDFs (maximum entropy). Multivariate Gaussianization and multivariate equalization are specific examples of this family, which break down the complexity of the original PDF through a set of elementary transforms that progressively remove the structure of the data. We demonstrate how this property of density destructive flows is connected to classical information theory, and how density destructors can be used to get more accurate estimates of information theoretic quantities. Experiments with total correlation and mutual information inmultivariate sets illustrate the ability of density destructors compared to competing methods. These results suggest that information theoretic measures may be an alternative optimization criteria when learning density destructive flows.