GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

TL;DR

GeoGuide introduces a hierarchical geometric framework for open-vocabulary 3D segmentation, integrating 3D models with semantic consistency modules to improve accuracy and robustness over existing 2D-dependent methods.

Contribution

It proposes a novel 3D segmentation framework that leverages pretrained 3D models and hierarchical geometry-semantic modules to address limitations of 2D-based approaches.

Findings

Outperforms existing methods on ScanNet v2, Matterport3D, and nuScenes datasets.

Effectively suppresses noise and preserves discriminative features through uncertainty-based superpoint distillation.

Enhances semantic consistency within instances and across objects using geometric priors and relation alignment.

Abstract

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.