GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Grounding

TL;DR

GroundFlow is a plug-in module that enhances 3D visual grounding by enabling effective temporal reasoning on sequential point cloud data, significantly improving accuracy in locating object sequences from text instructions.

Contribution

We introduce GroundFlow, a novel plug-in module for temporal reasoning in 3D visual grounding, which improves baseline methods and achieves state-of-the-art results in SG3D tasks.

Findings

GroundFlow improves 3DVG baseline accuracy by over 7% and 10%.

It outperforms pre-trained 3D large language models on SG3D benchmark.

Selective extraction of short-term and long-term information enhances temporal understanding.

Abstract

Sequential grounding in 3D point clouds (SG3D) refers to locating sequences of objects by following text instructions for a daily activity with detailed steps. Current 3D visual grounding (3DVG) methods treat text instructions with multiple steps as a whole, without extracting useful temporal information from each step. However, the instructions in SG3D often contain pronouns such as "it", "here" and "the same" to make language expressions concise. This requires grounding methods to understand the context and retrieve relevant information from previous steps to correctly locate object sequences. Due to the lack of an effective module for collecting related historical information, state-of-the-art 3DVG methods face significant challenges in adapting to the SG3D task. To fill this gap, we propose GroundFlow -- a plug-in module for temporal reasoning on 3D point cloud sequential grounding. Firstly, we demonstrate that integrating GroundFlow improves the task accuracy of 3DVG baseline methods by a large margin (+7.5\% and +10.2\%) in the SG3D benchmark, even outperforming a 3D large language model pre-trained on various datasets. Furthermore, we selectively extract both short-term and long-term step information based on its relevance to the current instruction, enabling GroundFlow to take a comprehensive view of historical information and maintain its temporal understanding advantage as step counts increase. Overall, our work introduces temporal reasoning capabilities to existing 3DVG models and achieves state-of-the-art performance in the SG3D benchmark across five datasets.