SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection

TL;DR

SparseLIF introduces a novel sparse multi-modality 3D object detector that leverages perspective priors, refined sampling, and uncertainty-aware fusion to achieve state-of-the-art results on nuScenes.

Contribution

The paper presents a fully sparse, end-to-end multi-modality 3D detector with three innovative components for improved performance and robustness.

Findings

Achieves state-of-the-art results on nuScenes dataset.

Outperforms all existing 3D object detectors.

Demonstrates robustness against sensor noise.

Abstract

Sparse 3D detectors have received significant attention since the query-based paradigm embraces low latency without explicit dense BEV feature construction. However, these detectors achieve worse performance than their dense counterparts. In this paper, we find the key to bridging the performance gap is to enhance the awareness of rich representations in two modalities. Here, we present a high-performance fully sparse detector for end-to-end multi-modality 3D object detection. The detector, termed SparseLIF, contains three key designs, which are (1) Perspective-Aware Query Generation (PAQG) to generate high-quality 3D queries with perspective priors, (2) RoI-Aware Sampling (RIAS) to further refine prior queries by sampling RoI features from each modality, (3) Uncertainty-Aware Fusion (UAF) to precisely quantify the uncertainty of each sensor modality and adaptively conduct final multi-modality fusion, thus achieving great robustness against sensor noises. By the time of paper submission, SparseLIF achieves state-of-the-art performance on the nuScenes dataset, ranking 1st on both validation set and test benchmark, outperforming all state-of-the-art 3D object detectors by a notable margin.