Visual Language Model based Cross-modal Semantic Communication Systems

TL;DR

This paper introduces a novel vision-language model-based cross-modal semantic communication system that enhances image transmission efficiency, robustness, and adaptability in dynamic environments by leveraging high-density semantics, memory mechanisms, and noise-aware attention.

Contribution

The paper proposes a new VLM-CSC system with three innovative components: CKB, MED, and NAM, addressing key challenges in existing ISC systems.

Findings

Improves semantic transmission efficiency by using high-density textual semantics.

Enhances robustness against semantic drift and noise through memory and attention mechanisms.

Validated effectiveness and adaptability through experimental simulations.

Abstract

Semantic Communication (SC) has emerged as a novel communication paradigm in recent years, successfully transcending the Shannon physical capacity limits through innovative semantic transmission concepts. Nevertheless, extant Image Semantic Communication (ISC) systems face several challenges in dynamic environments, including low semantic density, catastrophic forgetting, and uncertain Signal-to-Noise Ratio (SNR). To address these challenges, we propose a novel Vision-Language Model-based Cross-modal Semantic Communication (VLM-CSC) system. The VLM-CSC comprises three novel components: (1) Cross-modal Knowledge Base (CKB) is used to extract high-density textual semantics from the semantically sparse image at the transmitter and reconstruct the original image based on textual semantics at the receiver. The transmission of high-density semantics contributes to alleviating bandwidth pressure. (2) Memory-assisted Encoder and Decoder (MED) employ a hybrid long/short-term memory mechanism, enabling the semantic encoder and decoder to overcome catastrophic forgetting in dynamic environments when there is a drift in the distribution of semantic features. (3) Noise Attention Module (NAM) employs attention mechanisms to adaptively adjust the semantic coding and the channel coding based on SNR, ensuring the robustness of the CSC system. The experimental simulations validate the effectiveness, adaptability, and robustness of the CSC system.