Multi-variate correlation and mixtures of product measures

TL;DR

This paper extends the theory of total correlation and dual total correlation to general random variables and reveals that small TC or DTC implies the distribution is close to a product measure or a mixture of such measures, respectively.

Contribution

It generalizes the definitions of TC and DTC beyond finite variables and establishes new structural results linking small DTC to mixtures of product measures.

Findings

Small TC implies distribution is close to a product measure.

Small DTC implies distribution is a mixture of near-product measures.

Provides new theoretical insights into the structure of distributions with low correlation.

Abstract

Total correlation (`TC') and dual total correlation (`DTC') are two classical ways to quantify the correlation among an $n$-tuple of random variables. They both reduce to mutual information when $n=2$. The first part of this paper sets up the theory of TC and DTC for general random variables, not necessarily finite-valued. This generality has not been exposed in the literature before. The second part considers the structural implications when a joint distribution $\mu$ has small TC or DTC. If $\mathrm{TC}(\mu) = o(n)$, then $\mu$ is close to a product measure according to a suitable transportation metric: this follows directly from Marton's classical transportation-entropy inequality. If $\mathrm{DTC}(\mu) = o(n)$, then the structural consequence is more complicated: $\mu$ is a mixture of a controlled number of terms, most of them close to product measures in the transportation metric. This is the main new result of the paper.