HeCoFuse: Cross-Modal Complementary V2X Cooperative Perception with Heterogeneous Sensors

TL;DR

HeCoFuse is a novel framework for V2X cooperative perception that effectively fuses heterogeneous sensor data using hierarchical attention and adaptive modules, achieving state-of-the-art results across diverse sensor configurations.

Contribution

We introduce HeCoFuse, a unified adaptive fusion framework for heterogeneous sensors in V2X perception, addressing cross-modality misalignment and robustness issues.

Findings

Achieves 43.22% 3D mAP with full sensor setup, outperforming baseline.

Maintains high performance across nine sensor configurations.

First-place in CVPR 2025 DriveX challenge.

Abstract

Real-world Vehicle-to-Everything (V2X) cooperative perception systems often operate under heterogeneous sensor configurations due to cost constraints and deployment variability across vehicles and infrastructure. This heterogeneity poses significant challenges for feature fusion and perception reliability. To address these issues, we propose HeCoFuse, a unified framework designed for cooperative perception across mixed sensor setups where nodes may carry Cameras (C), LiDARs (L), or both. By introducing a hierarchical fusion mechanism that adaptively weights features through a combination of channel-wise and spatial attention, HeCoFuse can tackle critical challenges such as cross-modality feature misalignment and imbalanced representation quality. In addition, an adaptive spatial resolution adjustment module is employed to balance computational cost and fusion effectiveness. To enhance robustness across different configurations, we further implement a cooperative learning strategy that dynamically adjusts fusion type based on available modalities. Experiments on the real-world TUMTraf-V2X dataset demonstrate that HeCoFuse achieves 43.22% 3D mAP under the full sensor configuration (LC+LC), outperforming the CoopDet3D baseline by 1.17%, and reaches an even higher 43.38% 3D mAP in the L+LC scenario, while maintaining 3D mAP in the range of 21.74% to 43.38% across nine heterogeneous sensor configurations. These results, validated by our first-place finish in the CVPR 2025 DriveX challenge, establish HeCoFuse as the current state-of-the-art on TUM-Traf V2X dataset while demonstrating robust performance across diverse sensor deployments.