DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion

TL;DR

DecoratingFusion introduces a novel LiDAR-camera fusion network that combines point-level and feature-level fusion, utilizing hard associations from calibration matrices to improve 3D detection accuracy in autonomous driving.

Contribution

It innovatively integrates hard point-level associations with feature-level fusion, enhancing interpretability and detection performance over existing methods.

Findings

Outperforms state-of-the-art methods on KITTI and Waymo datasets.

Effectively combines hard and soft associations for improved 3D detection.

Demonstrates superior accuracy in autonomous driving scenarios.

Abstract

Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion.