Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

TL;DR

This paper introduces a novel geometry-only supervision framework for open-vocabulary 3D occupancy prediction in indoor scenes, leveraging Gaussian representations and a progressive sharpening technique to improve semantic and geometric understanding.

Contribution

It proposes a new Gaussian-based approach with a stabilization method and a progressive semantic sharpening schedule for better indoor occupancy prediction.

Findings

Achieves 59.50 IoU and 21.05 mIoU on Occ-ScanNet in open-vocabulary setting

Surpasses existing occupancy methods in IoU and prior open-vocabulary approaches in mIoU

Introduces a stabilization technique and a semantic sharpening schedule that improve indoor scene understanding

Abstract

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.