FAST3DIS: Feed-forward Anchored Scene Transformer for 3D Instance Segmentation

TL;DR

FAST3DIS is an end-to-end 3D instance segmentation method using a Transformer architecture that avoids clustering, improves efficiency, and maintains geometric priors for better scene understanding.

Contribution

The paper introduces a novel query-based Transformer architecture with 3D anchors and contrastive learning for efficient, accurate 3D instance segmentation without post-hoc clustering.

Findings

Achieves competitive accuracy on indoor 3D datasets.

Offers improved memory scalability and inference speed.

Effectively prevents query collisions with dual-level regularization.

Abstract

While recent feed-forward 3D reconstruction models provide a strong geometric foundation for scene understanding, extending them to 3D instance segmentation typically relies on a disjointed "lift-and-cluster" paradigm. Grouping dense pixel-wise embeddings via non-differentiable clustering scales poorly with the number of views and disconnects representation learning from the final segmentation objective. In this paper, we present a Feed-forward Anchored Scene Transformer for 3D Instance Segmentation (FAST3DIS), an end-to-end approach that effectively bypasses post-hoc clustering. We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone, adapted efficiently to learn instance-specific semantics while retaining its zero-shot geometric priors. We formulate a learned 3D anchor generator coupled with an anchor-sampling cross-attention mechanism for view-consistent 3D instance segmentation. By projecting 3D object queries directly into multi-view feature maps, our method samples context efficiently. Furthermore, we introduce a dual-level regularization strategy, that couples multi-view contrastive learning with a dynamically scheduled spatial overlap penalty to explicitly prevent query collisions and ensure precise instance boundaries. Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy with significantly improved memory scalability and inference speed over state-of-the-art clustering-based methods.