SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

TL;DR

SpaCeFormer is a fast, proposal-free 3D instance segmentation method that significantly outperforms prior approaches in speed and accuracy, enabling real-time applications in robotics and AR/VR.

Contribution

It introduces SpaCeFormer, a novel transformer-based model that predicts 3D instance masks directly without external proposals, and provides the largest open-vocabulary 3D segmentation dataset.

Findings

Runs at 0.14 seconds per scene, 2-3 orders faster than previous methods.

Achieves 21x higher mask recall than prior single-view pipelines.

Surpasses all prior methods on multiple datasets in zero-shot mAP.

Abstract

Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs at 0.14 seconds per scene, 2-3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches 21x higher mask recall than prior single-view pipelines (54.3% vs 2.5% at IoU > 0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8x improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.