VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System

TL;DR

VLF-MSC introduces a unified semantic communication system using vision-language features to efficiently transmit and generate both images and text, improving robustness and spectral efficiency over existing methods.

Contribution

The paper presents a novel unified system that encodes and transmits a single vision-language feature for both image and text generation, unlike prior modality-specific approaches.

Findings

Outperforms text-only and image-only baselines in semantic accuracy

Achieves higher robustness to channel noise

Reduces bandwidth requirements significantly

Abstract

We propose Vision-Language Feature-based Multimodal Semantic Communication (VLF-MSC), a unified system that transmits a single compact vision-language representation to support both image and text generation at the receiver. Unlike existing semantic communication techniques that process each modality separately, VLF-MSC employs a pre-trained vision-language model (VLM) to encode the source image into a vision-language semantic feature (VLF), which is transmitted over the wireless channel. At the receiver, a decoder-based language model and a diffusion-based image generator are both conditioned on the VLF to produce a descriptive text and a semantically aligned image. This unified representation eliminates the need for modality-specific streams or retransmissions, improving spectral efficiency and adaptability. By leveraging foundation models, the system achieves robustness to channel noise while preserving semantic fidelity. Experiments demonstrate that VLF-MSC outperforms text-only and image-only baselines, achieving higher semantic accuracy for both modalities under low SNR with significantly reduced bandwidth.