Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior

TL;DR

Diff-ICMH introduces a generative image compression framework that harmonizes machine and human vision by ensuring semantic fidelity and perceptual realism, enabling multiple tasks with a single codec and minimal bit rate overhead.

Contribution

The paper proposes Diff-ICMH, a novel generative compression method that combines semantic consistency and perceptual quality, supporting diverse tasks without task-specific tuning.

Findings

Outperforms existing methods in multiple tasks

Maintains high perceptual quality for human viewers

Supports diverse intelligent tasks with a single codec

Abstract

Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks. Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training. Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception. Code is available at: https://github.com/RuoyuFeng/Diff-ICMH.