ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding

TL;DR

ViGiL3D is a new dataset that provides linguistically diverse prompts for 3D visual grounding, revealing current models' limitations in understanding complex language patterns for real-world scene understanding.

Contribution

We introduce ViGiL3D, a diagnostic dataset with diverse linguistic prompts for 3D visual grounding, and analyze existing models' performance on challenging out-of-distribution language patterns.

Findings

Existing models struggle with complex, out-of-distribution prompts.

ViGiL3D reveals gaps in current 3D visual grounding methods.

Diverse language prompts improve evaluation of model robustness.

Abstract

3D visual grounding (3DVG) involves localizing entities in a 3D scene referred to by natural language text. Such models are useful for embodied AI and scene retrieval applications, which involve searching for objects or patterns using natural language descriptions. While recent works have focused on LLM-based scaling of 3DVG datasets, these datasets do not capture the full range of potential prompts which could be specified in the English language. To ensure that we are scaling up and testing against a useful and representative set of prompts, we propose a framework for linguistically analyzing 3DVG prompts and introduce Visual Grounding with Diverse Language in 3D (ViGiL3D), a diagnostic dataset for evaluating visual grounding methods against a diverse set of language patterns. We evaluate existing open-vocabulary 3DVG methods to demonstrate that these methods are not yet proficient in understanding and identifying the targets of more challenging, out-of-distribution prompts, toward real-world applications.