Multimodal generative semantic communication based on latent diffusion model

TL;DR

This paper presents mm-GESCO, a multimodal generative semantic communication framework that fuses visible and infrared data, achieving high compression and improved accuracy in environmental understanding tasks.

Contribution

The paper introduces a novel multimodal semantic communication framework using latent diffusion models and contrastive learning for data fusion and reconstruction.

Findings

Achieves up to 200x data compression ratio.

Outperforms existing semantic communication methods.

Enhances downstream task performance like classification and detection.

Abstract

In emergencies, the ability to quickly and accurately gather environmental data and command information, and to make timely decisions, is particularly critical. Traditional semantic communication frameworks, primarily based on a single modality, are susceptible to complex environments and lighting conditions, thereby limiting decision accuracy. To this end, this paper introduces a multimodal generative semantic communication framework named mm-GESCO. The framework ingests streams of visible and infrared modal image data, generates fused semantic segmentation maps, and transmits them using a combination of one-hot encoding and zlib compression techniques to enhance data transmission efficiency. At the receiving end, the framework can reconstruct the original multimodal images based on the semantic maps. Additionally, a latent diffusion model based on contrastive learning is designed to align different modal data within the latent space, allowing mm-GESCO to reconstruct latent features of any modality presented at the input. Experimental results demonstrate that mm-GESCO achieves a compression ratio of up to 200 times, surpassing the performance of existing semantic communication frameworks and exhibiting excellent performance in downstream tasks such as object classification and detection.