SGIFormer: Semantic-guided and Geometric-enhanced Interleaving Transformer for 3D Instance Segmentation

TL;DR

SGIFormer introduces a novel transformer-based approach for 3D instance segmentation, leveraging semantic-guided query initialization and geometric-enhanced decoding to improve accuracy and efficiency in large-scale 3D scenes.

Contribution

The paper proposes SGIFormer, combining semantic-guided query initialization and a geometric-enhanced interleaving transformer decoder for improved 3D instance segmentation.

Findings

Achieves state-of-the-art results on ScanNet V2 and ScanNet200 datasets.

Balances accuracy and efficiency effectively.

Demonstrates robustness on challenging high-fidelity benchmarks.

Abstract

In recent years, transformer-based models have exhibited considerable potential in point cloud instance segmentation. Despite the promising performance achieved by existing methods, they encounter challenges such as instance query initialization problems and excessive reliance on stacked layers, rendering them incompatible with large-scale 3D scenes. This paper introduces a novel method, named SGIFormer, for 3D instance segmentation, which is composed of the Semantic-guided Mix Query (SMQ) initialization and the Geometric-enhanced Interleaving Transformer (GIT) decoder. Specifically, the principle of our SMQ initialization scheme is to leverage the predicted voxel-wise semantic information to implicitly generate the scene-aware query, yielding adequate scene prior and compensating for the learnable query set. Subsequently, we feed the formed overall query into our GIT decoder to alternately refine instance query and global scene features for further capturing fine-grained information and reducing complex design intricacies simultaneously. To emphasize geometric property, we consider bias estimation as an auxiliary task and progressively integrate shifted point coordinates embedding to reinforce instance localization. SGIFormer attains state-of-the-art performance on ScanNet V2, ScanNet200 datasets, and the challenging high-fidelity ScanNet++ benchmark, striking a balance between accuracy and efficiency. The code, weights, and demo videos are publicly available at https://rayyoh.github.io/sgiformer.