SQ-GAN: Semantic Image Communications Using Masked Vector Quantization

TL;DR

SQ-GAN introduces a semantically driven image compression method that selectively encodes relevant features, outperforming existing schemes at low bit rates by optimizing for semantic and perceptual quality.

Contribution

The paper presents a novel semantically masked vector quantization approach integrated into GANs for task-oriented image compression, compatible with legacy systems.

Findings

Outperforms JPEG2000, BPG, and deep-learning methods in quality metrics.

Effective at extremely low compression rates.

Enhances semantic segmentation accuracy on reconstructed images.

Abstract

This work introduces Semantically Masked Vector Quantized Generative Adversarial Network (SQ-GAN), a novel approach integrating semantically driven image coding and vector quantization to optimize image compression for semantic/task-oriented communications. The method only acts on source coding and is fully compliant with legacy systems. The semantics is extracted from the image computing its semantic segmentation map using off-the-shelf software. A new specifically developed semantic-conditioned adaptive mask module (SAMM) selectively encodes semantically relevant features of the image. The relevance of the different semantic classes is task-specific, and it is incorporated in the training phase by introducing appropriate weights in the loss function. SQ-GAN outperforms state-of-the-art image compression schemes such as JPEG2000, BPG, and deep-learning based methods across multiple metrics, including perceptual quality and semantic segmentation accuracy on the reconstructed image, at extremely low compression rates.