Your Demands Deserve More Bits: Referring Semantic Image Compression at Ultra-low Bitrate

TL;DR

This paper introduces RSIC, a novel image compression framework that uses semantic guidance to improve fidelity of user-specified content at ultra-low bitrates, balancing realism and compression efficiency.

Contribution

The paper proposes a new RSIC framework with modules for semantic encoding and guided generative decoding, enhancing fidelity and flexibility in ultra-low bitrate image compression.

Findings

RSIC achieves superior compression efficiency on three datasets.

RSIC provides flexible, user-demanded content reconstruction.

RSIC maintains high semantic alignment at ultra-low bitrates.

Abstract

With the help of powerful generative models, Semantic Image Compression (SIC) has achieved impressive performance at ultra-low bitrate. However, due to coarse-grained visual-semantic alignment and inherent randomness, the reliability of SIC is seriously concerned for reconstructing completely different object instances, even they are semantically consistent with original images. To tackle this issue, we propose a novel Referring Semantic Image Compression (RSIC) framework to improve the fidelity of user-specified content while retaining extreme compression ratios. Specifically, RSIC consists of three modules: Global Description Encoding (GDE), Referring Guidance Encoding (RGE), and Guided Generative Decoding (GGD). GDE and RGE encode global semantic information and local features, respectively, while GGD handles the non-uniformly guided generative process based on the encoded information. In this way, our RSIC achieves flexible customized compression according to user demands, which better balance the local fidelity, global realism, semantic alignment, and bit overhead. Extensive experiments on three datasets verify the compression efficiency and flexibility of the proposed method.