CAM3DNet: Comprehensively mining the multi-scale features for 3D Object Detection with Multi-View Cameras

TL;DR

CAM3DNet is a novel framework for 3D object detection from multi-view images that effectively leverages multi-scale spatiotemporal features through three innovative modules, outperforming existing methods.

Contribution

The paper introduces CAM3DNet, which combines composite query, adaptive self-attention, and multi-scale hybrid sampling modules for improved multi-view 3D detection.

Findings

Outperforms existing camera-based 3D detection methods on nuScenes, Waymo, and Argoverse datasets.

Demonstrates effectiveness of the three proposed modules through extensive experiments.

Provides ablation studies on the individual contributions and computational costs of each module.

Abstract

Query-based 3D object detection methods using multi-view images often struggle to efficiently leverage dynamic multi-scale information, e.g., the relationship between the object features and the geometric of the queries are not sufficiently learned, directly exploring the multi-scale spatiotemporal features will pay too many costs. To address these challenges, we propose CAM3DNet, a novel sparse query-based framework which combines three new modules, composite query (CQ), adaptive self-attention (ASA), and multi-scale hybrid sampling (MSHS). First, the core idea in the CQ module is a multi-scale projection strategy to transform 2D queries into 3D space. Second, the ASA module learns the interactions between the spatiotemporal multi-scale queries. Third, the MSHS module uses the deformable attention mechanism to sample multi-scale object information by considering multi-scales queries, pyramid feature maps, and 2D-camera prior knowledge. The entire model employs a backbone network and a feature pyramid network (FPN) as the encoder, then introduces a YOLOX and a DepthNet as a ROI\_Head to produce CQ, and repeatedly utilizes ASA and MSHS as the decoder to gain detection features. Extensive experiments on the nuScenes, Waymo, and Argoverse benchmark datasets demonstrate the effectiveness of our CAM3DNet, and most existing camera-based 3D object detection methods are outperformed. Besides, we make comprehensive ablation studies to check the individual effect of CQ, ASA, and MSHS, as well as their cost of space and computation complexity.