Image Generation with Supervised Selection Based on Multimodal Features for Semantic Communications

TL;DR

This paper introduces a multimodal semantic communication framework that uses both image and text features to supervise image generation, improving fidelity and robustness over traditional single-modal approaches, especially in noisy conditions.

Contribution

The paper proposes a novel multimodal semantic communication system utilizing CNN and CLIP for feature extraction and a diffusion model for image generation, enhancing semantic fidelity and robustness.

Findings

Improved image transmission fidelity compared to existing systems.

Enhanced robustness in low SNR environments.

Effective multiuser extension maintaining high performance.

Abstract

Semantic communication (SemCom) has emerged as a promising technique for the next-generation communication systems, in which the generation at the receiver side is allowed with semantic features' recovery. However, the majority of existing research predominantly utilizes a singular type of semantic information, such as text, images, or speech, to supervise and choose the generated source signals, which may not sufficiently encapsulate the comprehensive and accurate semantic information, and thus creating a performance bottleneck. In order to bridge this gap, in this paper, we propose and investigate a SemCom framework using multimodal information to supervise the generated image. To be specific, in this framework, we first extract semantic features at both the image and text levels utilizing the Convolutional Neural Network (CNN) architecture and the Contrastive Language-Image Pre-Training (CLIP) model before transmission. Then, we employ a generative diffusion model at the receiver to generate multiple images. In order to ensure the accurate extraction and facilitate high-fidelity image reconstruction, we select the "best" image with the minimum reconstruction errors by taking both the aided image and text semantic features into account. We further extend multimodal semantic communication (MMSemCom) system to the multiuser scenario for orthogonal transmission. Experimental results demonstrate that the proposed framework can not only achieve the enhanced fidelity and robustness in image transmission compared with existing communication systems but also sustain a high performance in the low signal-to-noise ratio (SNR) conditions.