Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion

TL;DR

ResULIC is a novel ultra lowrate image compression framework that leverages semantic residual coding and a compression-aware diffusion model to significantly improve reconstruction quality and coding efficiency.

Contribution

It introduces Semantic Residual Coding and a Compression-aware Diffusion Model, enhancing image compression by better capturing semantic disparities and aligning diffusion processes with bitrate constraints.

Findings

Achieves 80.7% BD-rate saving in LPIPS

Achieves 66.3% BD-rate saving in FID

Outperforms state-of-the-art diffusion-based methods

Abstract

Existing multimodal large model-based image compression frameworks often rely on a fragmented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruction fidelity and coding efficiency. To address these challenges, we propose a residual-guided ultra lowrate image compression named ResULIC, which incorporates residual signals into both semantic retrieval and the diffusion-based generation process. Specifically, we introduce Semantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A perceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, we present the Compression-aware Diffusion Model (CDM), which establishes an optimal alignment between bitrates and diffusion time steps, improving compression-reconstruction synergy. Extensive experiments demonstrate the effectiveness of ResULIC, achieving superior objective and subjective performance compared to state-of-the-art diffusion-based methods with - 80.7%, -66.3% BD-rate saving in terms of LPIPS and FID. Project page is available at https: //njuvision.github.io/ResULIC/.