On the Entropy of Couplings

TL;DR

This paper explores the properties of Shannon information measures over restricted probability distributions, introduces the concept of minimum entropy coupling, and studies related pseudometrics, revealing computational hardness and connections to classical problems.

Contribution

It introduces the notion of minimum entropy coupling and analyzes its properties, linking it to computational complexity and information-theoretic measures.

Findings

Certain optimization problems are NP-hard.

Minimum entropy coupling is relevant in multiple contexts.

New pseudometrics relate to total variation and conditional entropy.

Abstract

In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be NP-hard, and their special cases are found to be essentially information-theoretic restatements of well-known computational problems, such as the SUBSET SUM and the 3-PARTITION. The notion of minimum entropy coupling is introduced and its relevance is demonstrated in information-theoretic, computational, and statistical contexts. Finally, a family of pseudometrics (on the space of discrete probability distributions) defined by these couplings is studied, in particular their relation to the total variation distance, and a new characterization of the conditional entropy is given.