Linking Modality Isolation in Heterogeneous Collaborative Perception

TL;DR

This paper introduces CodeAlign, a novel modality alignment framework for collaborative perception that effectively handles modality isolation without requiring co-occurrence data, significantly reducing training complexity and communication load.

Contribution

CodeAlign is the first co-occurrence-free alignment method that uses feature-code-feature translation with codebooks to align heterogeneous modalities in collaborative perception.

Findings

Requires only 8% of training parameters compared to prior methods.

Reduces communication load by 1024 times.

Achieves state-of-the-art perception performance on multiple datasets.

Abstract

Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.