Language-Guided Diffusion Model for Visual Grounding

TL;DR

This paper introduces LG-DVG, a diffusion-based iterative reasoning framework for visual grounding that refines object localization by denoising noisy boxes guided by language, outperforming existing single-step methods.

Contribution

It proposes a novel diffusion model approach for visual grounding, enabling iterative refinement of object boxes guided by language semantics, unlike prior one-step reasoning methods.

Findings

Outperforms existing methods on five datasets

Demonstrates effective iterative box refinement

Validates the generative diffusion approach for visual grounding

Abstract

Visual grounding (VG) tasks involve explicit cross-modal alignment, as semantically corresponding image regions are to be located for the language phrases provided. Existing approaches complete such visual-text reasoning in a single-step manner. Their performance causes high demands on large-scale anchors and over-designed multi-modal fusion modules based on human priors, leading to complicated frameworks that may be difficult to train and overfit to specific scenarios. Even worse, such once-for-all reasoning mechanisms are incapable of refining boxes continuously to enhance query-region matching. In contrast, in this paper, we formulate an iterative reasoning process by denoising diffusion modeling. Specifically, we propose a language-guided diffusion framework for visual grounding, LG-DVG, which trains the model to progressively reason queried object boxes by denoising a set of noisy boxes with the language guide. To achieve this, LG-DVG gradually perturbs query-aligned ground truth boxes to noisy ones and reverses this process step by step, conditional on query semantics. Extensive experiments for our proposed framework on five widely used datasets validate the superior performance of solving visual grounding, a cross-modal alignment task, in a generative way. The source codes are available at https://github.com/iQua/vgbase/tree/main/examples/DiffusionVG.