Distributed Image Compression with Multimodal Side Information at Extremely Low Bitrates

TL;DR

This paper introduces a multimodal distributed image compression framework that leverages side information through a diffusion-based decoder and feature masking to improve image quality at extremely low bitrates.

Contribution

It presents the first multimodal approach to distributed image compression, effectively utilizing side information for enhanced reconstruction quality.

Findings

Achieves state-of-the-art perceptual quality at < 0.1 bpp.

Effectively preserves fine details and global semantics.

Outperforms existing methods on KITTI Stereo and Cityscapes datasets.

Abstract

Distributed Image Compression (DIC) is crucial for multi-view transmission, especially when operating at extremely low bitrates (< 0.1 bpp). Its core challenge is effectively utilizing side information to achieve high-quality reconstruction under strict bitrate budgets. However, existing DIC approaches struggle to exploit global context and object-level details from side information, leading to local blurring and the loss of fine details in the reconstruction. To address these limitations, we propose a Multimodal DIC framework (MDIC), which, for the first time, leverages side information in a multimodal manner into the DIC paradigm, effectively preserving fine-grained local details and enhancing global perceptual quality in reconstructed images. Specifically, we introduce a text-to-image diffusion-based decoder conditioned on textual side information extracted from correlated images to capture shared global semantics. Moreover, we design a feature-mask generator, supervised by a multimodal fine-grained alignment task, to strengthen the exploitation of visual side information. The generated mask serves two purposes: first, it guides the extraction of fine-grained details from losslessly transmitted side information to preserve the semantic consistency of reconstructed details; second, it regulates the extraction of clustered feature representations from the quantized VQ-VAE embeddings, compensating for category information lost under the extreme compression of the primary image. Extensive experiments on the widely used KITTI Stereo and Cityscapes datasets demonstrate that MDIC achieves state-of-the-art perceptual quality at extremely low bitrates.