R2G: Reasoning to Ground in 3D Scenes

TL;DR

R2G introduces a neural symbolic model that grounds objects in 3D scenes through explicit scene graph reasoning, enhancing interpretability while maintaining competitive accuracy.

Contribution

The paper presents a novel reasoning-based approach for 3D object grounding that explicitly models scene graphs and attention transfer, improving interpretability over prior methods.

Findings

Achieves comparable accuracy to prior methods on Sr3D/Nr3D benchmarks.

Provides interpretable reasoning process for 3D object grounding.

Breaks new ground in interpretable 3D language grounding.

Abstract

We propose Reasoning to Ground (R2G), a neural symbolic model that grounds the target objects within 3D scenes in a reasoning manner. In contrast to prior works, R2G explicitly models the 3D scene with a semantic concept-based scene graph; recurrently simulates the attention transferring across object entities; thus makes the process of grounding the target objects with the highest probability interpretable. Specifically, we respectively embed multiple object properties within the graph nodes and spatial relations among entities within the edges, utilizing a predefined semantic vocabulary. To guide attention transferring, we employ learning or prompting-based methods to analyze the referential utterance and convert it into reasoning instructions within the same semantic space. In each reasoning round, R2G either (1) merges current attention distribution with the similarity between the instruction and embedded entity properties or (2) shifts the attention across the scene graph based on the similarity between the instruction and embedded spatial relations. The experiments on Sr3D/Nr3D benchmarks show that R2G achieves a comparable result with the prior works while maintaining improved interpretability, breaking a new path for 3D language grounding.