JEPA-MSAC: A Joint-Embedding Predictive Architecture for Multimodal Sensing-Assisted Communications

TL;DR

JEPA-MSAC introduces a self-supervised, multimodal predictive framework for wireless environments that supports multiple PHY tasks with low adaptation, enhancing transferability and efficiency.

Contribution

It proposes a novel joint-embedding predictive architecture that learns environment dynamics and cross-modal dependencies for multi-task wireless communication applications.

Findings

Supports accurate multi-task prediction with low adaptation cost

Latent state effectively captures environment dynamics and cross-modal dependencies

Ablation studies show the importance of pretraining setups

Abstract

Future wireless systems increasingly require predictive and transferable representations that can support multiple physical-layer (PHY) tasks under dynamic environments. However, most existing supervised learning-based methods are designed for a single task, which leads to high adaptation cost. To address this issue, we propose a joint-embedding predictive architecture for multimodal sensing-assisted communications (JEPA-MSAC), a self-supervised multimodal predictive representation learning framework for wireless environments. The proposed framework first maps multimodal sensing and communication measurements into a unified token space, and then pretrains a shared backbone using temporal block-masked JEPA to learn a predictive latent space that captures environment dynamics and cross-modal dependencies. After pretraining, the backbone is frozen and reused as a general future-feature generator, on top of which lightweight task heads are trained for localization, beam prediction, and received signal strength indicator (RSSI) prediction. Extensive experiments show the latent state supports accurate multi-task prediction with low adaptation cost. Additionally, ablation studies reveal its scaling behavior and the impact of key pretraining setups.