Universally Quantized Neural Compression

TL;DR

This paper introduces a method using universal quantization to align training and testing phases in neural compression, providing a practical, differentiable approach that bridges soft and hard quantization.

Contribution

It demonstrates that universal quantization can replace additive noise during training, eliminating phase mismatch and enabling efficient, practical neural compression.

Findings

Universal quantization can be used at test time to match training conditions.

The uniform noise channel implementation is computationally efficient and practical.

Quantization can be viewed as a limit of soft quantization applied to the noise channel.

Abstract

A popular approach to learning encoders for lossy compression is to use additive uniform noise during training as a differentiable approximation to test-time quantization. We demonstrate that a uniform noise channel can also be implemented at test time using universal quantization (Ziv, 1985). This allows us to eliminate the mismatch between training and test phases while maintaining a completely differentiable loss function. Implementing the uniform noise channel is a special case of the more general problem of communicating a sample, which we prove is computationally hard if we do not make assumptions about its distribution. However, the uniform special case is efficient as well as easy to implement and thus of great interest from a practical point of view. Finally, we show that quantization can be obtained as a limiting case of a soft quantizer applied to the uniform noise channel, bridging compression with and without quantization.