Singular Relative Entropy Coding with Bits-Back Rejection Sampling

TL;DR

This paper introduces the bits-back rejection sampler (BBRS), a practical and efficient relative entropy coding method that simplifies previous approaches and achieves near-optimal asymptotic redundancy for singular channels.

Contribution

The paper presents BBRS, a new relative entropy coding scheme combining bits-back coding and rejection sampling, improving simplicity and constants over prior methods.

Findings

BBRS achieves asymptotic efficiency similar to previous complex samplers.

BBRS can be implemented with standard relative entropy coding techniques.

The analysis shows BBRS has better constants and simpler analysis than prior work.

Abstract

A relative entropy code for a source $X \sim P_X$ is a stochastic code that encodes random samples from a prescribed $P_{Y \mid X}$ using as few bits as possible. A generalisation of entropy coding, it is a standard result that the minimum number of bits required to achieve this is at least the mutual information $I[X\,\Vert\,Y]$. However, a particularly fascinating feature of relative entropy coding compared to entropy coding is that, in general, this lower bound is only achievable to within an additional logarithmic factor. As such, an important research direction is to identify channels where we can reduce this gap. Sriramu and Wagner achieved such success by exhibiting a relative entropy code for so-called singular channels with sub-logarithmic asymptotic redundancy. However, their code is quite involved and, sadly, cannot be implemented in practice. In this paper, we construct the bits-back rejection sampler (BBRS), a relative entropy code that combines ideas from bits-back coding and (greedy) rejection sampling. Our analysis of BBRS reveals that the algorithm achieves the same asymptotic efficiency as Sriramu and Wagner's sampler, but with much simpler analysis and better constants. Moreover, BBRS can be implemented using standard relative entropy coding methods.