Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation

TL;DR

This paper introduces BFL, a novel transformer-based approach for 3D instance segmentation that enhances query initialization and maintains high recall across layers, leading to superior performance on multiple datasets.

Contribution

The paper proposes an Agent-Interpolation Initialization Module and a Hierarchical Query Fusion Decoder to improve query resilience and layer-wise object recall in 3D instance segmentation.

Findings

BFL outperforms existing methods on ScanNetV2, ScanNet200, ScanNet++, and S3DIS datasets.

The proposed modules effectively balance foreground coverage and content learning.

Deep layers maintain higher recall with the hierarchical query fusion approach.

Abstract

3D instance segmentation aims to predict a set of object instances in a scene and represent them as binary foreground masks with corresponding semantic labels. Currently, transformer-based methods are gaining increasing attention due to their elegant pipelines, reduced manual selection of geometric properties, and superior performance. However, transformer-based methods fail to simultaneously maintain strong position and content information during query initialization. Additionally, due to supervision at each decoder layer, there exists a phenomenon of object disappearance with the deepening of layers. To overcome these hurdles, we introduce Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation (BFL). Specifically, an Agent-Interpolation Initialization Module is designed to generate resilient queries capable of achieving a balance between foreground coverage and content learning. Additionally, a Hierarchical Query Fusion Decoder is designed to retain low overlap queries, mitigating the decrease in recall with the deepening of layers. Extensive experiments on ScanNetV2, ScanNet200, ScanNet++ and S3DIS datasets demonstrate the superior performance of BFL.