MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding

TL;DR

This paper introduces MAF, a framework that leverages caption-image datasets with weak supervision to improve phrase grounding by modeling phrase-object relevance with contrastive learning and visually-aware language representations.

Contribution

The paper proposes a novel multimodal alignment framework that enhances weakly-supervised phrase grounding using contrastive objectives and visually-aware language models, improving performance on Flickr30k.

Findings

Significant improvement over existing weakly-supervised methods.

Achieved a 5.56% boost over previous best unsupervised results.

Both the model and strategies contribute to strong performance.

Abstract

Phrase localization is a task that studies the mapping from textual phrases to regions of an image. Given difficulties in annotating phrase-to-object datasets at scale, we develop a Multimodal Alignment Framework (MAF) to leverage more widely-available caption-image datasets, which can then be used as a form of weak supervision. We first present algorithms to model phrase-object relevance by leveraging fine-grained visual representations and visually-aware language representations. By adopting a contrastive objective, our method uses information in caption-image pairs to boost the performance in weakly-supervised scenarios. Experiments conducted on the widely-adopted Flickr30k dataset show a significant improvement over existing weakly-supervised methods. With the help of the visually-aware language representations, we can also improve the previous best unsupervised result by 5.56%. We conduct ablation studies to show that both our novel model and our weakly-supervised strategies significantly contribute to our strong results.