Generative Semantic Coding for Ultra-Low Bitrate Visual Communication and Analysis

TL;DR

This paper introduces a novel method combining deep image compression and text-guided generation to enable accurate visual scene reconstruction at ultra-low bitrates, facilitating remote vision analysis and human interaction in bandwidth-constrained scenarios.

Contribution

It presents a new approach that integrates semantic text descriptions with deep image compression, achieving high-quality visual reconstruction at minimal bandwidth.

Findings

Achieves comparable image quality to existing methods at significantly lower bitrates.

Maintains vision analysis accuracy despite ultra-low bandwidth transmission.

Demonstrates effectiveness in challenging scenarios like deep space and battlefield environments.

Abstract

We consider the problem of ultra-low bit rate visual communication for remote vision analysis, human interactions and control in challenging scenarios with very low communication bandwidth, such as deep space exploration, battlefield intelligence, and robot navigation in complex environments. In this paper, we ask the following important question: can we accurately reconstruct the visual scene using only a very small portion of the bit rate in existing coding methods while not sacrificing the accuracy of vision analysis and performance of human interactions? Existing text-to-image generation models offer a new approach for ultra-low bitrate image description. However, they can only achieve a semantic-level approximation of the visual scene, which is far insufficient for the purpose of visual communication and remote vision analysis and human interactions. To address this important issue, we propose to seamlessly integrate image generation with deep image compression, using joint text and coding latent to guide the rectified flow models for precise generation of the visual scene. The semantic text description and coding latent are both encoded and transmitted to the decoder at a very small bit rate. Experimental results demonstrate that our method can achieve the same image reconstruction quality and vision analysis accuracy as existing methods while using much less bandwidth. The code will be released upon paper acceptance.