RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

TL;DR

RG-SAN introduces a rule-guided spatial awareness network that leverages spatial information and dependency rules to improve 3D referring expression segmentation, achieving state-of-the-art performance and robustness.

Contribution

The paper proposes a novel RG-SAN framework that uses solely spatial information and dependency rules for supervision, enhancing 3D segmentation accuracy and reasoning.

Findings

Achieved 5.1 points higher mIoU on ScanRefer benchmark.

Significantly improved robustness to spatial ambiguity.

Outperformed existing methods in 3D referring expression segmentation.

Abstract

3D Referring Expression Segmentation (3D-RES) aims to segment 3D objects by correlating referring expressions with point clouds. However, traditional approaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances. In this paper, we introduce a Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision. This approach enables the network to accurately depict the spatial relationships among all entities described in the text, thus enhancing the reasoning capabilities. The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. The TLM initially locates all mentioned instances and iteratively refines their positional information. The RWS strategy, acknowledging that only target objects have supervised positional information, employs dependency tree rules to precisely guide the core instance's positioning. Extensive testing on the ScanRefer benchmark has shown that RG-SAN not only establishes new performance benchmarks, with an mIoU increase of 5.1 points, but also exhibits significant improvements in robustness when processing descriptions with spatial ambiguity. All codes are available at https://github.com/sosppxo/RG-SAN.