LDRFusion: A LiDAR-Dominant multimodal refinement framework for 3D object detection

TL;DR

LDRFusion introduces a LiDAR-dominant two-stage refinement framework for 3D object detection that effectively combines LiDAR and pseudo point clouds, improving accuracy by addressing noise and leveraging modality reliability.

Contribution

The paper proposes a novel LiDAR-dominant multi-sensor fusion framework with hierarchical pseudo point residual encoding, enhancing 3D detection accuracy over existing methods.

Findings

Consistently outperforms previous methods on KITTI dataset

Effective in detecting challenging instances with pseudo point clouds

Improves local structure representation in pseudo points

Abstract

Existing LiDAR-Camera fusion methods have achieved strong results in 3D object detection. To address the sparsity of point clouds, previous approaches typically construct spatial pseudo point clouds via depth completion as auxiliary input and adopts a proposal-refinement framework to generate detection results. However, introducing pseudo points inevitably brings noise, potentially resulting in inaccurate predictions. Considering the differing roles and reliability levels of each modality, we propose LDRFusion, a novel Lidar-dominant two-stage refinement framework for multi-sensor fusion. The first stage soley relies on LiDAR to produce accurately localized proposals, followed by a second stage where pseudo point clouds are incorporated to detect challenging instances. The instance-level results from both stages are subsequently merged. To further enhance the representation of local structures in pseudo point clouds, we present a hierarchical pseudo point residual encoding module, which encodes neighborhood sets using both feature and positional residuals. Experiments on the KITTI dataset demonstrate that our framework consistently achieves strong performance across multiple categories and difficulty levels.