Does Vision-and-Language Pretraining Improve Lexical Grounding?

TL;DR

This paper investigates whether vision-and-language pretraining enhances linguistic representations by comparing multimodal models to text-only models, finding limited improvements and highlighting the need for further research.

Contribution

It provides a comparative analysis of semantic representations in VL and text-only models, revealing that VL pretraining does not significantly outperform text-only pretraining.

Findings

Multimodal models do not significantly outperform text-only models in linguistic tasks.

Analysis methods include clustering, probing, and question answering performance.

Future work is needed to determine the potential of multimodal pretraining for NLP.

Abstract

Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.