Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning

TL;DR

GroundingAgent introduces a training-free, agentic visual grounding framework that leverages pretrained models and iterative reasoning to achieve high zero-shot accuracy and interpretability in linking text to image regions.

Contribution

It presents a novel, training-free visual grounding method combining pretrained detectors and language models with iterative reasoning, enhancing generalization and interpretability.

Findings

Achieves 65.1% zero-shot accuracy on benchmarks

Substituting captions with queries yields ~90% accuracy

Operates without task-specific fine-tuning

Abstract

Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fine-tuning, limiting their ability to generalize effectively to novel or out-of-distribution scenarios. To address these limitations, we introduce GroundingAgent, a novel agentic visual grounding framework that operates without any task-specific fine-tuning. GroundingAgent employs a structured, iterative reasoning mechanism that integrates pretrained open-vocabulary object detectors, multimodal large language models (MLLMs), and large language models (LLMs) to progressively refine candidate regions through joint semantic and spatial analyses. Remarkably, GroundingAgent achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks (RefCOCO, RefCOCO+, RefCOCOg), entirely without fine-tuning. Furthermore, by substituting MLLM-generated captions with the original query texts, the accuracy at the selection stage alone reaches approximately 90 %, closely matching supervised performance and underscoring the critical role of LLM reasoning capabilities. GroundingAgent also offers strong interpretability, transparently illustrating each reasoning step and providing clear insights into its decision-making process.