UniMIC: Towards Universal Multi-modality Perceptual Image Compression

TL;DR

UniMIC introduces a universal multi-modality image compression framework that unifies rate-distortion-perception optimization across various codecs using cross-modality priors and a perception compensator, enhancing image quality especially at low bitrates.

Contribution

The paper proposes a novel universal framework that integrates multiple codecs and multi-modal priors for improved perceptual image compression, a significant advancement over existing single-codec approaches.

Findings

Achieves significant RDP improvements across traditional and learnable codecs.

Enhances perceptual quality of decoded images at ultra-low bitrates.

Utilizes cross-modality generative priors and a perception compensator for universal applicability.

Abstract

We present UniMIC, a universal multi-modality image compression framework, intending to unify the rate-distortion-perception (RDP) optimization for multiple image codecs simultaneously through excavating cross-modality generative priors. Unlike most existing works that need to design and optimize image codecs from scratch, our UniMIC introduces the visual codec repository, which incorporates amounts of representative image codecs and directly uses them as the basic codecs for various practical applications. Moreover, we propose multi-grained textual coding, where variable-length content prompt and compression prompt are designed and encoded to assist the perceptual reconstruction through the multi-modality conditional generation. In particular, a universal perception compensator is proposed to improve the perception quality of decoded images from all basic codecs at the decoder side by reusing text-assisted diffusion priors from stable diffusion. With the cooperation of the above three strategies, our UniMIC achieves a significant improvement of RDP optimization for different compression codecs, e.g., traditional and learnable codecs, and different compression costs, e.g., ultra-low bitrates. The code will be available in https://github.com/Amygyx/UniMIC .