STELLAR: Scaling 3D Perception Large Models for Autonomous Driving

TL;DR

This paper introduces STELLAR, a large-scale 3D perception model for autonomous driving that leverages extensive data and model scaling to achieve state-of-the-art results.

Contribution

The paper systematically analyzes the impact of scale on autonomous driving perception systems and develops a large-scale model with diverse sensor inputs.

Findings

Empirical scaling trends link model size, data, and compute to performance.

STELLAR outperforms previous methods on the Waymo Open Dataset.

Large-scale training significantly advances perception capabilities.

Abstract

Model scaling has demonstrated remarkable success through large-scale training on diverse datasets. It remains an open question whether the same paradigm would apply to autonomous driving perception systems due to unique challenges, such as fusing heterogeneous sensor data and the need for sophisticated 3D spatial understanding. To bridge this gap, we present a comprehensive study on systematically analyzing the impact of scale on these systems. We develop our STELLAR model based on Sparse Window Transformer, by extending the input modalities to include LiDAR, radar, camera, and map prior. We train the model on a large-scale dataset of 50 million driving examples with up to 500 million parameters. Our large-scale experiments reveal empirical scaling trends that connect model performance to model size, data, and compute. The resulting model establishes a new state-of-the-art on the Waymo Open Dataset challenge, outperforming prior arts by a large margin. Our work demonstrates that large-scale training is a highly promising path for advancing the capabilities of perception models for autonomous driving.