Learned Compression of Encoding Distributions

TL;DR

This paper introduces a dynamic encoding distribution approach for learned compression models that adapts to each input, improving compression efficiency and reducing computational complexity compared to static methods.

Contribution

It proposes a method to estimate and transmit an input-specific encoding distribution, addressing the amortization gap in learned compression models.

Findings

Achieves a -7.10% BD-rate gain on Kodak dataset.

Reduces computational complexity by an order of magnitude.

Outperforms static distribution methods in adaptability and efficiency.

Abstract

The entropy bottleneck introduced by Ball\'e et al. is a common component used in many learned compression models. It encodes a transformed latent representation using a static distribution whose parameters are learned during training. However, the actual distribution of the latent data may vary wildly across different inputs. The static distribution attempts to encompass all possible input distributions, thus fitting none of them particularly well. This unfortunate phenomenon, sometimes known as the amortization gap, results in suboptimal compression. To address this issue, we propose a method that dynamically adapts the encoding distribution to match the latent data distribution for a specific input. First, our model estimates a better encoding distribution for a given input. This distribution is then compressed and transmitted as an additional side-information bitstream. Finally, the decoder reconstructs the encoding distribution and uses it to decompress the corresponding latent data. Our method achieves a Bj{\o}ntegaard-Delta (BD)-rate gain of -7.10% on the Kodak test dataset when applied to the standard fully-factorized architecture. Furthermore, considering computational complexity, the transform used by our method is an order of magnitude cheaper in terms of Multiply-Accumulate (MAC) operations compared to related side-information methods such as the scale hyperprior.