ImLiDAR: Cross-Sensor Dynamic Message Propagation Network for 3D Object Detection

TL;DR

ImLiDAR introduces a novel cross-sensor fusion approach for 3D object detection that effectively combines multi-scale features from LiDAR and camera data, improving detection accuracy and robustness.

Contribution

The paper presents a new fusion paradigm with a dynamic message propagation module and a set prediction detector, advancing cross-sensor 3D detection methods.

Findings

Outperforms 23 state-of-the-art methods on KITTI and SUN-RGBD datasets.

Provides clear visual and numerical improvements in 3D detection.

Offers a flexible, detachable set-based detection head.

Abstract

LiDAR and camera, as two different sensors, supply geometric (point clouds) and semantic (RGB images) information of 3D scenes. However, it is still challenging for existing methods to fuse data from the two cross sensors, making them complementary for quality 3D object detection (3OD). We propose ImLiDAR, a new 3OD paradigm to narrow the cross-sensor discrepancies by progressively fusing the multi-scale features of camera Images and LiDAR point clouds. ImLiDAR enables to provide the detection head with cross-sensor yet robustly fused features. To achieve this, two core designs exist in ImLiDAR. First, we propose a cross-sensor dynamic message propagation module to combine the best of the multi-scale image and point features. Second, we raise a direct set prediction problem that allows designing an effective set-based detector to tackle the inconsistency of the classification and localization confidences, and the sensitivity of hand-tuned hyperparameters. Besides, the novel set-based detector can be detachable and easily integrated into various detection networks. Comparisons on both the KITTI and SUN-RGBD datasets show clear visual and numerical improvements of our ImLiDAR over twenty-three state-of-the-art 3OD methods.