SiMO: Single-Modality-Operable Multimodal Collaborative Perception

TL;DR

SiMO is a novel collaborative perception framework that maintains performance across modalities by adaptively fusing features and ensuring modality independence, even during sensor failures.

Contribution

Introduces SiMO, a multimodal perception method that handles modal failures and preserves modality-specific features through adaptive fusion and a new training strategy.

Findings

Effective alignment of multimodal features.

Maintains performance during modal failures.

Preserves modality-specific features.

Abstract

Collaborative perception integrates multi-agent perspectives to enhance the sensing range and overcome occlusion issues. While existing multimodal approaches leverage complementary sensors to improve performance, they are highly prone to failure--especially when a key sensor like LiDAR is unavailable. The root cause is that feature fusion leads to semantic mismatches between single-modality features and the downstream modules. This paper addresses this challenge for the first time in the field of collaborative perception, introducing Single-Modality-Operable Multimodal Collaborative Perception (SiMO). By adopting the proposed Length-Adaptive Multi-Modal Fusion (LAMMA), SiMO can adaptively handle remaining modal features during modal failures while maintaining consistency of the semantic space. Additionally, leveraging the innovative "Pretrain-Align-Fuse-RD" training strategy, SiMO addresses the issue of modality competition--generally overlooked by existing methods--ensuring the independence of each individual modality branch. Experiments demonstrate that SiMO effectively aligns multimodal features while simultaneously preserving modality-specific features, enabling it to maintain optimal performance across all individual modalities. The implementation details can be found in https://github.com/dempsey-wen/SiMO.