Rejection Sampling is Optimal for Relative Entropy Coding

TL;DR

This paper proves that rejection sampling is optimal for relative entropy coding, establishing tight bounds on the coding rate using the functional information measure, and improving upon classical mutual information bounds.

Contribution

The paper introduces the ring toss code, achieving the tight lower bound based on functional information, and refines the understanding of coding rates in relative entropy coding.

Findings

Rejection sampling achieves the tight lower bound of functional information plus a small constant.

Classical mutual information bounds are within a small additive gap for general channels.

The results recover and extend classical source coding theorems and second-order redundancy results.

Abstract

In relative entropy coding, a sender aims to design a stochastic code such that, on input $X \sim P_X$, the receiver can generate a sample $Y \sim P_{Y \mid X}$. It is a standard result that (1) this requires at least $I(X; Y)$ bits, (2) the lower bound is achievable within a logarithmic gap, and (3) this gap cannot be reduced in general. The necessity of the gap suggests that the mutual information is not the correct information measure to quantify the rate of relative entropy coding. A potential alternative emerged in the work of Flamich et al. (2025), who proved a tighter lower bound of $I_F(X \to Y)$, a quantity we call the functional information. In this paper, we show that this lower bound is tight by constructing the ring toss code, an encoding method for rejection sampling which uses at most $I_F(X \to Y) + \log e$ bits. For the trivial channel $Y = X$, our result recovers the noiseless source coding theorem within a small constant. For a general channel, it implies that the classical mutual information lower bound is achievable within $\log(I(X; Y) + 1) + 2.45$ bits in general and within $1.45$ bits for singular channels, which are both the tightest bounds of their kind to date. Moreover, our one-shot result also recovers Sriramu and Wagner's asymptotic results on the second-order redundancy of relative entropy codes.