GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splatting

TL;DR

GS-Occ3D introduces a scalable vision-only framework for occupancy reconstruction using Gaussian splatting and scene decomposition, enabling effective large-scale auto-labeling for autonomous driving without LiDAR data.

Contribution

It presents a novel Octree-based Gaussian Surfel method for explicit occupancy modeling, decomposes scenes into static and dynamic parts, and demonstrates superior results on large urban datasets.

Findings

Achieves state-of-the-art geometry reconstruction on Waymo dataset.

Effective vision-only occupancy labels improve downstream models.

Demonstrates strong zero-shot generalization to nuScenes dataset.

Abstract

Occupancy is crucial for autonomous driving, providing essential geometric priors for perception and planning. However, existing methods predominantly rely on LiDAR-based occupancy annotations, which limits scalability and prevents leveraging vast amounts of potential crowdsourced data for auto-labeling. To address this, we propose GS-Occ3D, a scalable vision-only framework that directly reconstructs occupancy. Vision-only occupancy reconstruction poses significant challenges due to sparse viewpoints, dynamic scene elements, severe occlusions, and long-horizon motion. Existing vision-based methods primarily rely on mesh representation, which suffer from incomplete geometry and additional post-processing, limiting scalability. To overcome these issues, GS-Occ3D optimizes an explicit occupancy representation using an Octree-based Gaussian Surfel formulation, ensuring efficiency and scalability. Additionally, we decompose scenes into static background, ground, and dynamic objects, enabling tailored modeling strategies: (1) Ground is explicitly reconstructed as a dominant structural element, significantly improving large-area consistency; (2) Dynamic vehicles are separately modeled to better capture motion-related occupancy patterns. Extensive experiments on the Waymo dataset demonstrate that GS-Occ3D achieves state-of-the-art geometry reconstruction results. By curating vision-only binary occupancy labels from diverse urban scenes, we show their effectiveness for downstream occupancy models on Occ3D-Waymo and superior zero-shot generalization on Occ3D-nuScenes. It highlights the potential of large-scale vision-based occupancy reconstruction as a new paradigm for scalable auto-labeling. Project Page: https://gs-occ3d.github.io/