OpenGround: Active Cognition-based Reasoning for Open-World 3D Visual Grounding

TL;DR

OpenGround introduces a zero-shot framework with an Active Cognition-based Reasoning module that enhances 3D visual grounding in open-world scenarios by dynamically expanding the model's understanding beyond predefined object categories.

Contribution

The paper proposes OpenGround, a novel open-world 3D visual grounding method with an Active Cognition-based Reasoning module that overcomes the limitations of pre-defined object lookup tables.

Findings

Achieves competitive performance on Nr3D

State-of-the-art on ScanRefer

17.6% improvement on OpenTarget

Abstract

3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes. Existing methods rely on a pre-defined Object Lookup Table (OLT) to query Visual Language Models (VLMs) for reasoning about object locations, which limits the applications in scenarios with undefined or unforeseen targets. To address this problem, we present OpenGround, a novel zero-shot framework for open-world 3D visual grounding. Central to OpenGround is the Active Cognition-based Reasoning (ACR) module, which is designed to overcome the fundamental limitation of pre-defined OLTs by progressively augmenting the cognitive scope of VLMs. The ACR module performs human-like perception of the target via a cognitive task chain and actively reasons about contextually relevant objects, thereby extending VLM cognition through a dynamically updated OLT. This allows OpenGround to function with both pre-defined and open-world categories. We also propose a new dataset named OpenTarget, which contains over 7000 object-description pairs to evaluate our method in open-world scenarios. Extensive experiments demonstrate that OpenGround achieves competitive performance on Nr3D, state-of-the-art on ScanRefer, and delivers a substantial 17.6% improvement on OpenTarget. Project Page at https://why-102.github.io/openground.io/.