Learning Visually Grounded Sentence Representations

TL;DR

This paper presents models trained on image captioning data to create grounded sentence representations, demonstrating improved performance on NLP tasks and analyzing the benefits of grounding and learned embeddings.

Contribution

The authors introduce a supervised grounding approach for sentence encoding that enhances NLP task performance and outperforms non-grounded embeddings.

Findings

Grounded sentence encoder performs well on COCO caption and image retrieval.

Transferred grounded encoder improves NLP task performance.

Grounded word embeddings outperform non-grounded ones.

Abstract

We introduce a variety of models, trained on a supervised image captioning corpus to predict the image features for a given caption, to perform sentence representation grounding. We train a grounded sentence encoder that achieves good performance on COCO caption and image retrieval and subsequently show that this encoder can successfully be transferred to various NLP tasks, with improved performance over text-only models. Lastly, we analyze the contribution of grounding, and show that word embeddings learned by this system outperform non-grounded ones.