Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints

TL;DR

This paper introduces C3VG, a two-stage multi-task visual grounding framework that enhances localization and segmentation accuracy by enforcing consistency constraints and leveraging pre-trained multimodal models.

Contribution

The paper proposes a novel coarse-to-fine architecture with explicit consistency constraints and multimodal pre-training to improve multi-task visual grounding performance.

Findings

Significantly outperforms state-of-the-art methods on RefCOCO, RefCOCO+, and RefCOCOg datasets.

Effectively enforces cross-task consistency through novel loss functions.

Leverages pre-trained visual-linguistic models to address understanding limitations.

Abstract

Multi-task visual grounding involves the simultaneous execution of localization and segmentation in images based on textual expressions. The majority of advanced methods predominantly focus on transformer-based multimodal fusion, aiming to extract robust multimodal representations. However, ambiguity between referring expression comprehension (REC) and referring image segmentation (RIS) is error-prone, leading to inconsistencies between multi-task predictions. Besides, insufficient multimodal understanding directly contributes to biased target perception. To overcome these challenges, we propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture ($\text{C}^3\text{VG}$), which integrates implicit and explicit modeling approaches within a two-stage framework. Initially, query and pixel decoders are employed to generate preliminary detection and segmentation outputs, a process referred to as the Rough Semantic Perception (RSP) stage. These coarse predictions are subsequently refined through the proposed Mask-guided Interaction Module (MIM) and a novel explicit bidirectional consistency constraint loss to ensure consistent representations across tasks, which we term the Refined Consistency Interaction (RCI) stage. Furthermore, to address the challenge of insufficient multimodal understanding, we leverage pre-trained models based on visual-linguistic fusion representations. Empirical evaluations on the RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate the efficacy and soundness of $\text{C}^3\text{VG}$, which significantly outperforms state-of-the-art REC and RIS methods by a substantial margin. Code and model will be available at \url{https://github.com/Dmmm1997/C3VG}.