Geometric-aware Pretraining for Vision-centric 3D Object Detection

TL;DR

This paper introduces GAPretrain, a geometric-aware pretraining framework that enhances vision-based 3D object detection by effectively leveraging spatial and structural cues from LiDAR and images, improving performance on autonomous driving benchmarks.

Contribution

We propose a novel pretraining method that incorporates geometric and structural cues using a unified BEV representation to improve view transformation and spatial feature extraction in 3D detection.

Findings

Achieves 46.2 mAP and 55.5 NDS on nuScenes val set with BEVFormer.

Demonstrates improved performance across various backbones and view transformations.

Serves as a flexible plug-and-play solution for state-of-the-art detectors.

Abstract

Multi-camera 3D object detection for autonomous driving is a challenging problem that has garnered notable attention from both academia and industry. An obstacle encountered in vision-based techniques involves the precise extraction of geometry-conscious features from RGB images. Recent approaches have utilized geometric-aware image backbones pretrained on depth-relevant tasks to acquire spatial information. However, these approaches overlook the critical aspect of view transformation, resulting in inadequate performance due to the misalignment of spatial knowledge between the image backbone and view transformation. To address this issue, we propose a novel geometric-aware pretraining framework called GAPretrain. Our approach incorporates spatial and structural cues to camera networks by employing the geometric-rich modality as guidance during the pretraining phase. The transference of modal-specific attributes across different modalities is non-trivial, but we bridge this gap by using a unified bird's-eye-view (BEV) representation and structural hints derived from LiDAR point clouds to facilitate the pretraining process. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. Our experiments demonstrate the effectiveness and generalization ability of the proposed method. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively. We also conduct experiments on various image backbones and view transformations to validate the efficacy of our approach. Code will be released at https://github.com/OpenDriveLab/BEVPerception-Survey-Recipe.