Breaking the Finite-Sample Barrier in Entropy Coupling

TL;DR

This paper introduces a new entropy coupling concept that captures how dependence among observations can eliminate residual uncertainty with finite samples, advancing understanding in distribution matching and randomness extraction.

Contribution

It formalizes the minimum list entropy coupling allowing dependence among variables, characterizes conditions for zero residual uncertainty, and provides algorithms for computation.

Findings

Dependent observations can eliminate residual uncertainty exactly after finitely many samples.

Zero entropy is achieved with O(log(1/P_min)) observations under mild support assumptions.

A greedy algorithm with monotone guarantees approximates the minimum list entropy coupling.

Abstract

Dependence among marginally constrained observations can break a finite-sample barrier. To formalize this phenomenon, we introduce the \emph{minimum list entropy coupling} $H(P\|Q_1,\dots,Q_m)$, the minimum conditional entropy $H(X|Y_1,\dots,Y_m)$ over all joint distributions with prescribed discrete marginals $X\sim P$ and $Y_i\sim Q_i$. Unlike classical formulations based on independent observations, our model allows $Y_1,\dots,Y_m$ to be arbitrarily dependent while keeping each marginal fixed. This enlarged coupling space reveals a sharp dichotomy: independent observations reduce residual uncertainty exponentially, whereas dependent observations can eliminate it exactly after finitely many samples. We characterize this zero-entropy regime through necessary and sufficient conditions and give concrete structural criteria under which it occurs. In particular, under mild support assumptions, zero entropy is achieved with $O(\log(1/P_{\min}))$ observations, where $P_{\min}$ is the minimum nonzero mass of $P$. We also develop a greedy algorithm with monotone approximation guarantees for computing $H(P\|Q_1,\dots,Q_m)$. Finally, we show that the same framework formalizes finite-sample limits in distribution-matching representation learning and randomness extraction, where zero entropy corresponds to exact recovery and exact extraction.