Invertible Low-Divergence Coding

TL;DR

This paper introduces an invertible encoding method with a one-to-many mapping and low-rate resolution code, achieving near-optimal information rates with exponentially decreasing divergence.

Contribution

It proposes a novel invertible encoder with a one-to-many mapping and develops algorithms that approach the target distribution's entropy with minimal divergence.

Findings

Achieves exponentially decreasing I-divergence with increasing block length.

Provides algorithms for mapping design based on likelihood ordering.

Attains information rates close to the entropy of the target distribution.

Abstract

Several applications in communication, control, and learning require approximating target distributions to within small informational divergence (I-divergence). The additional requirement of invertibility usually leads to using encoders that are one-to-one mappings, also known as distribution matchers. However, even the best one-to-one encoders have I-divergences that grow logarithmically with the block length in general. To improve performance, an encoder is proposed that has an invertible one-to-many mapping and a low-rate resolution code. Two algorithms are developed to design the mapping by assigning strings in either a most-likely first or least-likely first order. Both algorithms give information rates approaching the entropy of the target distribution with exponentially decreasing I-divergence and with vanishing resolution rate in the block length.