On the Feasibility and Opportunity of Autoregressive 3D Object Detection

TL;DR

AutoReg3D introduces an autoregressive, sequence-generation approach to LiDAR 3D object detection, eliminating the need for hand-crafted components and enabling new learning paradigms.

Contribution

It presents AutoReg3D, a novel autoregressive detection framework that models 3D detection as sequence generation, improving flexibility and performance without traditional proposal methods.

Findings

Achieves competitive nuScenes performance without anchors or NMS.

Demonstrates compatibility across diverse point-cloud backbones.

Enables integration of language-model techniques like reinforcement learning.

Abstract

LiDAR-based 3D object detectors typically rely on proposal heads with hand-crafted components like anchor assignment and non-maximum suppression (NMS), complicating training and limiting extensibility. We present AutoReg3D, an autoregressive 3D detector that casts detection as sequence generation. Given point-cloud features, AutoReg3D emits objects in a range-causal (near-to-far) order and encodes each object as a short, discrete-token sequence consisting of its center, size, orientation, velocity, and class. This near-to-far ordering mirrors LiDAR geometry--near objects occlude far ones but not vice versa--enabling straightforward teacher forcing during training and autoregressive decoding at test time. AutoReg3D is compatible across diverse point-cloud or backbones and attains competitive nuScenes performance without anchors or NMS. Beyond parity, the sequential formulation unlocks language-model advances for 3D perception, including GRPO-style reinforcement learning for task-aligned objectives. These results position autoregressive decoding as a viable, flexible alternative for LiDAR-based detection and open a path to importing modern sequence-modeling tools into 3D perception.