Generative Latent Coding for Ultra-Low Bitrate Image Compression

TL;DR

This paper introduces a Generative Latent Coding architecture that performs image compression in the latent space of a VQ-VAE, achieving high-quality results at ultra-low bitrates and enabling applications like restoration and style transfer.

Contribution

It proposes a novel latent space transform coding approach with a categorical hyper module and code-prediction supervision, improving compression efficiency and semantic consistency.

Findings

Maintains high visual quality at less than 0.04 bpp for natural images.

Achieves same FID as MS-ILLM with 45% fewer bits on CLIC2020.

Enables applications like image restoration and style transfer.

Abstract

Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.