Cross-Modal Semantic Communication for Heterogeneous Collaborative Perception

TL;DR

This paper introduces a cross-modal semantic communication framework that enables heterogeneous autonomous vehicles to share and fuse perceptual data effectively, improving collaborative perception especially under challenging conditions.

Contribution

It proposes a novel method to unify diverse sensor data into a common semantic space for better inter-vehicle communication and perception enhancement.

Findings

Significantly improves perception accuracy over existing methods.

Enhances robustness in low SNR environments.

Facilitates effective data fusion across different sensor modalities.

Abstract

Collaborative perception, an emerging paradigm in autonomous driving, has been introduced to mitigate the limitations of single-vehicle systems, such as limited sensor range and occlusion. To improve the robustness of inter-vehicle data sharing, semantic communication has recently further been integrated into collaborative perception systems to enhance overall performance. However, practical deployment of such systems is challenged by the heterogeneity of sensors across different connected autonomous vehicles (CAVs). This diversity in perceptual data complicates the design of a unified communication framework and impedes the effective fusion of shared information. To address this challenge, we propose a novel cross-modal semantic communication (CMSC) framework to facilitate effective collaboration among CAVs with disparate sensor configurations. Specifically, the framework first transforms heterogeneous perceptual features from different sensor modalities into a unified and standardized semantic space. Subsequently, encoding, transmission, and decoding are performed within this semantic space, enabling seamless and effective information fusion. Extensive experiments demonstrate that CMSC achieves significantly stronger perception performance than existing methods, particularly in low signal-to-noise ratio (SNR) regimes.