ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

TL;DR

ResVG is a novel model that improves visual grounding accuracy in complex scenes with multiple similar objects by integrating semantic priors and relation-sensitive data augmentation.

Contribution

The paper introduces ResVG, which enhances relation and semantic understanding in visual grounding through semantic prior injection and relation-aware data augmentation.

Findings

Significant performance improvements on five datasets.

Enhanced understanding of fine-grained semantics and spatial relations.

Better localization accuracy in multi-instance scenarios.

Abstract

Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at https://github.com/minghangz/ResVG.