Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

TL;DR

This paper introduces a soft-to-hard vector quantization method for end-to-end learning of compressible representations, achieving competitive results in image and neural network compression by smoothly transitioning from continuous to discrete representations during training.

Contribution

The authors propose a novel soft relaxation of quantization and entropy that is annealed to discrete values, enabling end-to-end training for compression tasks.

Findings

Achieves state-of-the-art results in image compression

Effective in neural network compression

Provides a unified approach for different compression tasks

Abstract

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.