Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving

TL;DR

This paper introduces Occ3D, a large-scale benchmark for 3D occupancy prediction in autonomous driving, along with a new dataset, label generation pipeline, and a novel Coarse-to-Fine Occupancy network that outperforms baselines.

Contribution

It provides the first large-scale 3D occupancy prediction benchmarks based on Waymo and nuScenes datasets, a comprehensive label generation pipeline, and a new model demonstrating superior performance.

Findings

The Occ3D benchmarks enable standardized evaluation of 3D occupancy prediction.

The label generation pipeline produces dense, visibility-aware labels for scenes.

The CTF-Occ model achieves state-of-the-art results on the benchmarks.

Abstract

Robotic perception requires the modeling of both 3D geometry and semantics. Existing methods typically focus on estimating 3D bounding boxes, neglecting finer geometric details and struggling to handle general, out-of-vocabulary objects. 3D occupancy prediction, which estimates the detailed occupancy states and semantics of a scene, is an emerging task to overcome these limitations. To support 3D occupancy prediction, we develop a label generation pipeline that produces dense, visibility-aware labels for any given scene. This pipeline comprises three stages: voxel densification, occlusion reasoning, and image-guided voxel refinement. We establish two benchmarks, derived from the Waymo Open Dataset and the nuScenes Dataset, namely Occ3D-Waymo and Occ3D-nuScenes benchmarks. Furthermore, we provide an extensive analysis of the proposed dataset with various baseline models. Lastly, we propose a new model, dubbed Coarse-to-Fine Occupancy (CTF-Occ) network, which demonstrates superior performance on the Occ3D benchmarks. The code, data, and benchmarks are released at https://tsinghua-mars-lab.github.io/Occ3D/.