Fully Sparse Fusion for 3D Object Detection

TL;DR

This paper introduces a fully sparse fusion framework for 3D object detection that effectively combines image and LiDAR data, achieving state-of-the-art accuracy and significantly faster inference for long-range perception.

Contribution

It presents a novel uniform query-based fusion approach within a fully sparse architecture, integrating 2D instance segmentation with LiDAR data for improved long-range 3D detection.

Findings

State-of-the-art results on nuScenes and Argoverse 2 datasets

Inference speed 2.7 times faster than existing methods

Effective long-range perception with fully sparse architecture

Abstract

Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View (BEV) feature maps. However, the cost of such BEV feature maps is quadratic to the detection range, making it not suitable for long-range detection. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture. Particularly, utilizing instance queries, our framework integrates the well-studied 2D instance segmentation into the LiDAR side, which is parallel to the 3D instance segmentation part in the fully sparse detector. This design achieves a uniform query-based fusion framework in both the 2D and 3D sides while maintaining the fully sparse characteristic. Extensive experiments showcase state-of-the-art results on the widely used nuScenes dataset and the long-range Argoverse 2 dataset. Notably, the inference speed of the proposed method under the long-range LiDAR perception setting is 2.7 $\times$ faster than that of other state-of-the-art multimodal 3D detection methods. Code will be released at \url{https://github.com/BraveGroup/FullySparseFusion}.