Accelerating Relative Entropy Coding with Space Partitioning

TL;DR

This paper introduces a space partitioning-based relative entropy coding scheme that significantly accelerates encoding times and improves compression efficiency, making REC more practical for neural compression tasks.

Contribution

It proposes a novel REC method using space partitioning, reducing runtime and enabling handling of higher KL divergence scenarios than previous methods.

Findings

Successfully handles REC with $D_{KL}$ three times larger than prior methods.

Reduces bitrate by 5-15% in VAE-based lossless compression on MNIST.

Improves practicality of REC for neural compression.

Abstract

Relative entropy coding (REC) algorithms encode a random sample following a target distribution $Q$, using a coding distribution $P$ shared between the sender and receiver. Sadly, general REC algorithms suffer from prohibitive encoding times, at least on the order of $2^{D_{\text{KL}}[Q||P]}$, and faster algorithms are limited to very specific settings. This work addresses this issue by introducing a REC scheme utilizing space partitioning to reduce runtime in practical scenarios. We provide theoretical analyses of our method and demonstrate its effectiveness with both toy examples and practical applications. Notably, our method successfully handles REC tasks with $D_{\text{KL}}[Q||P]$ about three times greater than what previous methods can manage, and reduces the bitrate by approximately 5-15% in VAE-based lossless compression on MNIST and INR-based lossy compression on CIFAR-10, compared to previous methods, significantly improving the practicality of REC for neural compression.