Contrastive Learning for Weakly Supervised Phrase Grounding

TL;DR

This paper introduces a contrastive learning approach for weakly supervised phrase grounding, leveraging mutual information maximization and language model-guided negatives to improve accuracy in associating image regions with caption words.

Contribution

It proposes a novel contrastive learning framework that uses language model-guided negative sampling to enhance weakly supervised phrase grounding performance.

Findings

Achieves approximately 10% accuracy gain over random negatives.

Improves Flickr30K Entities accuracy to 76.7%.

Demonstrates effectiveness of mutual information maximization in vision-language tasks.

Abstract

Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a $\sim10\%$ absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of $5.7\%$ to achieve $76.7\%$ accuracy on Flickr30K Entities benchmark.