Learning Visual Grounding from Generative Vision and Language Model

TL;DR

This paper demonstrates that generative vision-language models can be prompted to generate large-scale, high-quality visual grounding datasets, significantly improving zero-shot performance on referring expression tasks.

Contribution

The authors introduce a method to leverage generative VLMs for creating extensive visual grounding datasets with purely model-generated queries, enhancing zero-shot transfer capabilities.

Findings

Constructed a dataset with 500K images and 16M referring expressions.

Achieved state-of-the-art zero-shot results on RefCOCO benchmarks.

Showed that generative VLMs can effectively produce high-quality grounding data.

Abstract

Visual grounding tasks aim to localize image regions based on natural language references. In this work, we explore whether generative VLMs predominantly trained on image-text data could be leveraged to scale up the text annotation of visual grounding data. We find that grounding knowledge already exists in generative VLM and can be elicited by proper prompting. We thus prompt a VLM to generate object-level descriptions by feeding it object regions from existing object detection datasets. We further propose attribute modeling to explicitly capture the important object attributes, and spatial relation modeling to capture inter-object relationship, both of which are common linguistic pattern in referring expression. Our constructed dataset (500K images, 1M objects, 16M referring expressions) is one of the largest grounding datasets to date, and the first grounding dataset with purely model-generated queries and human-annotated objects. To verify the quality of this data, we conduct zero-shot transfer experiments to the popular RefCOCO benchmarks for both referring expression comprehension (REC) and segmentation (RES) tasks. On both tasks, our model significantly outperform the state-of-the-art approaches without using human annotated visual grounding data. Our results demonstrate the promise of generative VLM to scale up visual grounding in the real world. Code and models will be released.