SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos

TL;DR

SparseBEV introduces a fully sparse 3D object detection method that outperforms dense methods by using adaptive mechanisms in BEV and image spaces, achieving state-of-the-art results efficiently.

Contribution

The paper proposes SparseBEV, a novel sparse detector with adaptive features that significantly improves 3D detection performance over dense methods.

Findings

Achieves 67.5 NDS on nuScenes test split.

Maintains real-time inference at 23.5 FPS.

Outperforms dense detectors in accuracy.

Abstract

Camera-based 3D object detection in BEV (Bird's Eye View) space has drawn great attention over the past few years. Dense detectors typically follow a two-stage pipeline by first constructing a dense BEV feature and then performing object detection in BEV space, which suffers from complex view transformations and high computation cost. On the other side, sparse detectors follow a query-based paradigm without explicit dense BEV feature construction, but achieve worse performance than the dense counterparts. In this paper, we find that the key to mitigate this performance gap is the adaptability of the detector in both BEV and image space. To achieve this goal, we propose SparseBEV, a fully sparse 3D object detector that outperforms the dense counterparts. SparseBEV contains three key designs, which are (1) scale-adaptive self attention to aggregate features with adaptive receptive field in BEV space, (2) adaptive spatio-temporal sampling to generate sampling locations under the guidance of queries, and (3) adaptive mixing to decode the sampled features with dynamic weights from the queries. On the test split of nuScenes, SparseBEV achieves the state-of-the-art performance of 67.5 NDS. On the val split, SparseBEV achieves 55.8 NDS while maintaining a real-time inference speed of 23.5 FPS. Code is available at https://github.com/MCG-NJU/SparseBEV.