Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels

TL;DR

This paper introduces Ground-V, a scalable dataset and method for training vision-language models to perform pixel-level grounding of complex instructions, significantly improving accuracy on multiple benchmarks.

Contribution

We propose a knowledge distillation approach to automatically generate high-quality instruction-response pairs linked to pixel annotations, enabling effective training of grounding models without extensive human labeling.

Findings

Models trained on Ground-V show 4.4% accuracy improvement on LISA.

Achieves 7.9% higher accuracy on PSALM across six benchmarks.

Sets new state-of-the-art results on RefCOCO/+/g datasets.

Abstract

This work presents a simple yet effective workflow for automatically scaling instruction-following data to elicit pixel-level grounding capabilities of VLMs under complex instructions. In particular, we address five critical real-world challenges in text-instruction-based grounding: hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. By leveraging knowledge distillation from a pre-trained teacher model, our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations, minimizing the need for costly human annotation. The resulting dataset, Ground-V, captures rich object localization knowledge and nuanced pixel-level referring expressions. Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks. Specifically, incorporating Ground-V during training directly achieves an average accuracy boost of 4.4% for LISA and a 7.9% for PSALM across six benchmarks on the gIoU metric. It also sets new state-of-the-art results on standard benchmarks such as RefCOCO/+/g. Notably, on gRefCOCO, we achieve an N-Acc of 83.3%, exceeding the previous state-of-the-art by more than 20%.