Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs

TL;DR

This paper introduces SDComp, a novel image compression method that leverages Large Multimodal Models to encode images based on semantic importance, optimizing for machine understanding rather than human perception.

Contribution

The paper proposes a new semantic-aware image coding framework that uses LMMs to inform compression based on downstream task relevance, a significant shift from traditional methods.

Findings

Supports diverse vision tasks with structured bitstreams

Achieves better task performance compared to state-of-the-art codecs

Provides flexible and semantically meaningful image reconstructions

Abstract

We present a new image compression paradigm to achieve ``intelligently coding for machine'' by cleverly leveraging the common sense of Large Multimodal Models (LMMs). We are motivated by the evidence that large language/multimodal models are powerful general-purpose semantics predictors for understanding the real world. Different from traditional image compression typically optimized for human eyes, the image coding for machines (ICM) framework we focus on requires the compressed bitstream to more comply with different downstream intelligent analysis tasks. To this end, we employ LMM to \textcolor{red}{tell codec what to compress}: 1) first utilize the powerful semantic understanding capability of LMMs w.r.t object grounding, identification, and importance ranking via prompts, to disentangle image content before compression, 2) and then based on these semantic priors we accordingly encode and transmit objects of the image in order with a structured bitstream. In this way, diverse vision benchmarks including image classification, object detection, instance segmentation, etc., can be well supported with such a semantically structured bitstream. We dub our method ``\textit{SDComp}'' for ``\textit{S}emantically \textit{D}isentangled \textit{Comp}ression'', and compare it with state-of-the-art codecs on a wide variety of different vision tasks. SDComp codec leads to more flexible reconstruction results, promised decoded visual quality, and a more generic/satisfactory intelligent task-supporting ability.