Reliability-Driven LiDAR-Camera Fusion for Robust 3D Object Detection

TL;DR

ReliFusion is a novel LiDAR-camera fusion framework that enhances 3D object detection robustness by dynamically assessing sensor reliability and integrating multi-modal data in BEV space, especially under sensor failure conditions.

Contribution

The paper introduces ReliFusion, a new fusion framework with modules for reliability assessment and dynamic information balancing, improving robustness over existing methods.

Findings

Outperforms state-of-the-art methods on nuScenes dataset.

Maintains high detection accuracy under sensor malfunctions.

Enhances robustness in limited field-of-view scenarios.

Abstract

Accurate and robust 3D object detection is essential for autonomous driving, where fusing data from sensors like LiDAR and camera enhances detection accuracy. However, sensor malfunctions such as corruption or disconnection can degrade performance, and existing fusion models often struggle to maintain reliability when one modality fails. To address this, we propose ReliFusion, a novel LiDAR-camera fusion framework operating in the bird's-eye view (BEV) space. ReliFusion integrates three key components: the Spatio-Temporal Feature Aggregation (STFA) module, which captures dependencies across frames to stabilize predictions over time; the Reliability module, which assigns confidence scores to quantify the dependability of each modality under challenging conditions; and the Confidence-Weighted Mutual Cross-Attention (CW-MCA) module, which dynamically balances information from LiDAR and camera modalities based on these confidence scores. Experiments on the nuScenes dataset show that ReliFusion significantly outperforms state-of-the-art methods, achieving superior robustness and accuracy in scenarios with limited LiDAR fields of view and severe sensor malfunctions.