Divide and Conquer: Improving Multi-Camera 3D Perception with 2D Semantic-Depth Priors and Input-Dependent Queries

TL;DR

This paper introduces SDTR, an input-aware Transformer framework that leverages semantic and depth priors to improve multi-camera 3D perception tasks like object detection and BEV segmentation, achieving state-of-the-art results.

Contribution

The paper proposes a novel S-D Encoder and Prior-guided Query Builder that explicitly model and incorporate semantic and depth priors, enhancing Transformer-based 3D perception.

Findings

Achieves state-of-the-art performance on nuScenes and Lyft benchmarks.

Effectively models semantic and depth priors to improve 3D perception accuracy.

Enhances input-awareness of queries for better learning capacity.

Abstract

3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.