Communication Complexity of Exact Sampling under R\'enyi Information

TL;DR

This paper investigates the fundamental limits of exact sampling under exponential communication costs, establishing bounds related to Re9nyi divergence and demonstrating the asymptotic optimality of noncausal sampling.

Contribution

It provides tight upper and lower bounds on the Campbell cost of exact sampling using Re9nyi divergence, characterizes asymptotic optimality, and compares causal versus noncausal sampling performance.

Findings

Lower bound on Campbell cost grows with Re9nyi divergence.

Upper bound matches lower bound within b15-10 bits in numerical examples.

Noncausal samplers outperform causal ones asymptotically under exponential cost.

Abstract

We study the problem of exact sampling under an exponential communication cost, specifically Campbell's average codeword length $L(t)$ of order $t$, and R\'enyi's entropy. We provide a lower bound on the Campbell cost of exact sampling that grows approximately as $D_{1/\alpha}(P||Q)$, the R\'enyi divergence of order $1/\alpha$, with $\alpha = \frac{1}{1+t}$. Using the Poisson functional representation of Li and El Gamal, we prove an upper bound on $L(t)$ whose leading R\'enyi divergence term has order within $\epsilon$ of that of the lower bound. Our results reduce to the bounds of Harsha et al. as $\alpha \to 1$. We also provide numerical examples comparing the bounds in the cases of normal and Laplacian distributions, demonstrating that the upper and lower bounds are typically within 5-10 bits of each other. Our results characterize exactly the optimal asymptotic Campbell cost $L(t)$ per sample as the number of independent and identically distributed (i.i.d.) samples grows to infinity. We show that under the exponential cost, any causal sampler performs strictly worse asymptotically than noncausal samplers. This contrasts with the case of expected message length, where both causal and noncausal samplers have the same optimal asymptotic cost.