RoScenes: A Large-scale Multi-view 3D Dataset for Roadside Perception

TL;DR

RoScenes is a large-scale multi-view roadside perception dataset designed to advance vision-centric Bird's Eye View methods for complex traffic scenes, featuring extensive annotations and a novel annotation pipeline.

Contribution

The paper introduces RoScenes, the largest multi-view roadside perception dataset, and proposes RoBEV, a new BEV method with feature-guided position embedding, improving performance without extra computation.

Findings

RoScenes contains 21.13M 3D annotations over 64,000 m^2.

Current BEV methods underperform due to large perception areas and sensor variation.

RoBEV outperforms state-of-the-art methods significantly.

Abstract

We introduce RoScenes, the largest multi-view roadside perception dataset, which aims to shed light on the development of vision-centric Bird's Eye View (BEV) approaches for more challenging traffic scenes. The highlights of RoScenes include significantly large perception area, full scene coverage and crowded traffic. More specifically, our dataset achieves surprising 21.13M 3D annotations within 64,000 $m^2$. To relieve the expensive costs of roadside 3D labeling, we present a novel BEV-to-3D joint annotation pipeline to efficiently collect such a large volume of data. After that, we organize a comprehensive study for current BEV methods on RoScenes in terms of effectiveness and efficiency. Tested methods suffer from the vast perception area and variation of sensor layout across scenes, resulting in performance levels falling below expectations. To this end, we propose RoBEV that incorporates feature-guided position embedding for effective 2D-3D feature assignment. With its help, our method outperforms state-of-the-art by a large margin without extra computational overhead on validation set. Our dataset and devkit will be made available at https://github.com/xiaosu-zhu/RoScenes.