DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection

TL;DR

DepthFusion introduces a depth-aware feature fusion strategy for LiDAR-camera 3D detection, adaptively weighting modalities based on depth information to improve accuracy and robustness across datasets.

Contribution

This work is the first to incorporate depth-aware weighting in multi-modal feature fusion for 3D object detection, enhancing performance and robustness.

Findings

Outperforms previous state-of-the-art methods on nuScenes and KITTI datasets.

Demonstrates increased robustness to data corruptions on nuScenes-C.

Effectively balances modality contributions based on depth information.

Abstract

State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we are the first to observe that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DepthFusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-GFusion module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-LFusion module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes and KITTI datasets demonstrate that our DepthFusion method surpasses previous state-of-the-art methods. Moreover, our DepthFusion is more robust to various kinds of corruptions, outperforming previous methods on the nuScenes-C dataset.