VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders

TL;DR

VGDiffZero demonstrates that pre-trained text-to-image diffusion models can be effectively used for zero-shot visual grounding without additional training, leveraging a novel region-scoring method.

Contribution

The paper introduces VGDiffZero, a zero-shot visual grounding framework that applies pre-trained diffusion models directly to discriminative tasks without fine-tuning.

Findings

Achieves strong zero-shot visual grounding performance on RefCOCO datasets.

Introduces a comprehensive region-scoring method considering global and local contexts.

Demonstrates the effectiveness of generative diffusion models for discriminative visual tasks.

Abstract

Large-scale text-to-image diffusion models have shown impressive capabilities for generative tasks by leveraging strong vision-language alignment from pre-training. However, most vision-language discriminative tasks require extensive fine-tuning on carefully-labeled datasets to acquire such alignment, with great cost in time and computing resources. In this work, we explore directly applying a pre-trained generative diffusion model to the challenging discriminative task of visual grounding without any fine-tuning and additional training dataset. Specifically, we propose VGDiffZero, a simple yet effective zero-shot visual grounding framework based on text-to-image diffusion models. We also design a comprehensive region-scoring method considering both global and local contexts of each isolated proposal. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that VGDiffZero achieves strong performance on zero-shot visual grounding. Our code is available at https://github.com/xuyang-liu16/VGDiffZero.