Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion

TL;DR

This paper introduces SFD, a multi-modal framework that enhances 3D detection by using depth completion to generate pseudo point clouds, employing a novel fusion strategy, synchronized augmentation, and a specialized feature extractor, achieving top results on KITTI.

Contribution

The paper proposes a new multi-modal 3D detection framework with a grid-wise attentive fusion strategy, synchronized data augmentation, and a specialized feature extractor for pseudo point clouds, improving detection accuracy.

Findings

Achieves highest score on KITTI 3D detection leaderboard.

Introduces a novel 3D-GAF fusion strategy for better multi-modal integration.

Demonstrates effectiveness of pseudo point clouds with depth completion.

Abstract

Current LiDAR-only 3D detection methods inevitably suffer from the sparsity of point clouds. Many multi-modal methods are proposed to alleviate this issue, while different representations of images and point clouds make it difficult to fuse them, resulting in suboptimal performance. In this paper, we present a novel multi-modal framework SFD (Sparse Fuse Dense), which utilizes pseudo point clouds generated from depth completion to tackle the issues mentioned above. Different from prior works, we propose a new RoI fusion strategy 3D-GAF (3D Grid-wise Attentive Fusion) to make fuller use of information from different types of point clouds. Specifically, 3D-GAF fuses 3D RoI features from the couple of point clouds in a grid-wise attentive way, which is more fine-grained and more precise. In addition, we propose a SynAugment (Synchronized Augmentation) to enable our multi-modal framework to utilize all data augmentation approaches tailored to LiDAR-only methods. Lastly, we customize an effective and efficient feature extractor CPConv (Color Point Convolution) for pseudo point clouds. It can explore 2D image features and 3D geometric features of pseudo point clouds simultaneously. Our method holds the highest entry on the KITTI car 3D object detection leaderboard, demonstrating the effectiveness of our SFD. Codes are available at https://github.com/LittlePey/SFD.