Class-agnostic 3D Segmentation by Granularity-Consistent Automatic 2D Mask Tracking

TL;DR

This paper introduces a novel 3D segmentation method that maintains temporal consistency in 2D mask tracking and employs curriculum learning to improve 3D pseudo-label quality, achieving state-of-the-art results.

Contribution

It proposes a granularity-consistent 2D mask tracking technique combined with a three-stage curriculum learning framework for improved 3D segmentation.

Findings

Achieved state-of-the-art performance on standard benchmarks.

Generated more consistent and accurate 3D segmentations.

Demonstrated robustness in open-vocabulary scenarios.

Abstract

3D instance segmentation is an important task for real-world applications. To avoid costly manual annotations, existing methods have explored generating pseudo labels by transferring 2D masks from foundation models to 3D. However, this approach is often suboptimal since the video frames are processed independently. This causes inconsistent segmentation granularity and conflicting 3D pseudo labels, which degrades the accuracy of final segmentation. To address this, we introduce a Granularity-Consistent automatic 2D Mask Tracking approach that maintains temporal correspondences across frames, eliminating conflicting pseudo labels. Combined with a three-stage curriculum learning framework, our approach progressively trains from fragmented single-view data to unified multi-view annotations, ultimately globally coherent full-scene supervision. This structured learning pipeline enables the model to progressively expose to pseudo-labels of increasing consistency. Thus, we can robustly distill a consistent 3D representation from initially fragmented and contradictory 2D priors. Experimental results demonstrated that our method effectively generated consistent and accurate 3D segmentations. Furthermore, the proposed method achieved state-of-the-art results on standard benchmarks and open-vocabulary ability.