PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum

TL;DR

PGOV3D introduces a two-stage curriculum learning framework that leverages multi-view images, large language models, and auxiliary modules to improve open-vocabulary 3D semantic segmentation, achieving competitive results on standard benchmarks.

Contribution

The paper proposes a novel Partial-to-Global curriculum with a two-stage training strategy and multi-modal supervision for open-vocabulary 3D segmentation.

Findings

Effective partial scene pre-training with dense semantic labels.

Improved segmentation accuracy on ScanNet, ScanNet200, and S3DIS.

Robust cross-view feature consistency enforcement.

Abstract

Existing open-vocabulary 3D semantic segmentation methods typically supervise 3D segmentation models by merging text-aligned features (e.g., CLIP) extracted from multi-view images onto 3D points. However, such approaches treat multi-view images merely as intermediaries for transferring open-vocabulary information, overlooking their rich semantic content and cross-view correspondences, which limits model effectiveness. To address this, we propose PGOV3D, a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation. The key innovation lies in a two-stage training strategy. In the first stage, we pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry. These partial point clouds are derived from multi-view RGB-D inputs via pixel-wise depth projection. To enable open-vocabulary learning, we leverage a multi-modal large language model (MLLM) and a 2D segmentation foundation model to generate open-vocabulary labels for each viewpoint, offering rich and aligned supervision. An auxiliary inter-frame consistency module is introduced to enforce feature consistency across varying viewpoints and enhance spatial understanding. In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex. We aggregate the partial vocabularies associated with each scene and generate pseudo labels using the pre-trained model, effectively bridging the semantic gap between dense partial observations and large-scale 3D environments. Extensive experiments on ScanNet, ScanNet200, and S3DIS benchmarks demonstrate that PGOV3D achieves competitive performance in open-vocabulary 3D semantic segmentation.