Improved Visual Grounding through Self-Consistent Explanations

TL;DR

This paper introduces SelfEQ, a weakly-supervised method that enhances visual grounding in vision-language models by ensuring self-consistent explanations through paraphrase-based finetuning, leading to improved localization accuracy.

Contribution

The paper proposes a novel self-consistent explanation strategy using paraphrases and finetuning, significantly improving visual grounding performance without box annotations.

Findings

Improved accuracy on Flickr30k, ReferIt, and RefCOCO+ datasets.

Enhanced localization quality of gradient-based explanation methods.

Effective augmentation of vocabulary and object localization through self-consistency.

Abstract

Vision-and-language models trained to match images with text can be combined with visual explanation methods to point to the locations of specific objects in an image. Our work shows that the localization --"grounding"-- abilities of these models can be further improved by finetuning for self-consistent visual explanations. We propose a strategy for augmenting existing text-image datasets with paraphrases using a large language model, and SelfEQ, a weakly-supervised strategy on visual explanation maps for paraphrases that encourages self-consistency. Specifically, for an input textual phrase, we attempt to generate a paraphrase and finetune the model so that the phrase and paraphrase map to the same region in the image. We posit that this both expands the vocabulary that the model is able to handle, and improves the quality of the object locations highlighted by gradient-based visual explanation methods (e.g. GradCAM). We demonstrate that SelfEQ improves performance on Flickr30k, ReferIt, and RefCOCO+ over a strong baseline method and several prior works. Particularly, comparing to other methods that do not use any type of box annotations, we obtain 84.07% on Flickr30k (an absolute improvement of 4.69%), 67.40% on ReferIt (an absolute improvement of 7.68%), and 75.10%, 55.49% on RefCOCO+ test sets A and B respectively (an absolute improvement of 3.74% on average).