Zero-Shot Semantic Communication with Multimodal Foundation Models

TL;DR

This paper introduces SemCLIP, a zero-shot semantic communication framework using foundation models to transmit universal semantic tokens, significantly improving efficiency, robustness, and task generalization in low bandwidth and noisy conditions.

Contribution

The paper presents SemCLIP, a novel zero-shot SemCom system leveraging CLIP for universal semantic token transmission and a prompt learning mechanism for robustness, advancing flexible and efficient semantic communication.

Findings

41% improvement in zero-shot performance at low SNR

Over 50-fold bandwidth reduction compared to image transmission

Enhanced robustness through prompt adaptation

Abstract

Most existing semantic communication (SemCom) systems use deep joint source-channel coding (DeepJSCC) to encode task-specific semantics in a goal-oriented manner. However, their reliance on predefined tasks and datasets significantly limits their flexibility and generalizability in practical deployments. Multi-modal foundation models provide a promising solution by generating universal semantic tokens. Inspired by this, we introduce SemCLIP, a zero-shot SemCom framework leveraging the contrastive language-image pre-training (CLIP) model. By transmitting CLIP-generated image tokens instead of raw images, SemCLIP enables efficient SemCom under low bandwidth and challenging channel conditions, facilitating diverse downstream tasks and zero-shot applications. Specifically, we propose a DeepJSCC scheme for efficient CLIP token encoding. To mitigate potential degradation caused by compression and channel noise, a multi-modal transmission-aware prompt learning mechanism is designed at the receiver, which adapts prompts based on transmission quality, enhancing system robustness and channel adaptability. Simulation results demonstrate that SemCLIP outperforms the baselines, achieving a $41\%$ improvement in zero-shot performance at low signal-to-noise ratios. Meanwhile, SemCLIP reduces bandwidth usage by more than $50$-fold compared to alternative image transmission methods, demonstrating the potential of foundation models towards a generalized, task-agnostic SemCom solution.