RayFormer: Improving Query-Based Multi-Camera 3D Object Detection via Ray-Centric Strategies

TL;DR

RayFormer introduces a ray-centric approach to multi-camera 3D object detection, aligning query initialization and feature extraction with camera optics to improve detection accuracy and performance.

Contribution

The paper proposes a novel ray-inspired query initialization and feature extraction method that enhances multi-camera 3D detection accuracy by reducing feature ambiguity.

Findings

Achieves 55.5% mAP on nuScenes dataset

Attains 63.3% NDS, outperforming previous methods

Demonstrates effectiveness of ray-centric strategies in 3D detection

Abstract

The recent advances in query-based multi-camera 3D object detection are featured by initializing object queries in the 3D space, and then sampling features from perspective-view images to perform multi-round query refinement. In such a framework, query points near the same camera ray are likely to sample similar features from very close pixels, resulting in ambiguous query features and degraded detection accuracy. To this end, we introduce RayFormer, a camera-ray-inspired query-based 3D object detector that aligns the initialization and feature extraction of object queries with the optical characteristics of cameras. Specifically, RayFormer transforms perspective-view image features into bird's eye view (BEV) via the lift-splat-shoot method and segments the BEV map to sectors based on the camera rays. Object queries are uniformly and sparsely initialized along each camera ray, facilitating the projection of different queries onto different areas in the image to extract distinct features. Besides, we leverage the instance information of images to supplement the uniformly initialized object queries by further involving additional queries along the ray from 2D object detection boxes. To extract unique object-level features that cater to distinct queries, we design a ray sampling method that suitably organizes the distribution of feature sampling points on both images and bird's eye view. Extensive experiments are conducted on the nuScenes dataset to validate our proposed ray-inspired model design. The proposed RayFormer achieves superior performance of 55.5% mAP and 63.3% NDS, respectively.