AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting

TL;DR

AutoOcc introduces an automated, vision-guided pipeline for high-quality 3D semantic occupancy annotation that reduces manual effort and improves performance in complex scenes.

Contribution

The paper presents AutoOcc, a novel framework combining vision-language models and Gaussian splatting for automatic, open-ended 3D semantic occupancy annotation without human labels.

Findings

Outperforms existing automated annotation methods.

Effective in static and dynamic scenarios.

Enables open-ended semantic occupancy auto-labeling.

Abstract

Obtaining high-quality 3D semantic occupancy from raw sensor data remains an essential yet challenging task, often requiring extensive manual labeling. In this work, we propose AutoOcc, a vision-centric automated pipeline for open-ended semantic occupancy annotation that integrates differentiable Gaussian splatting guided by vision-language models. We formulate the open-ended semantic 3D occupancy reconstruction task to automatically generate scene occupancy by combining attention maps from vision-language models and foundation vision models. We devise semantic-aware Gaussians as intermediate geometric descriptors and propose a cumulative Gaussian-to-voxel splatting algorithm that enables effective and efficient occupancy annotation. Our framework outperforms existing automated occupancy annotation methods without human labels. AutoOcc also enables open-ended semantic occupancy auto-labeling, achieving robust performance in both static and dynamically complex scenarios.