What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge

TL;DR

This paper introduces evaluation tasks to measure visual commonsense knowledge in language models and finds that multimodal training does not significantly enhance this knowledge compared to unimodal models trained on visual text data.

Contribution

It presents new benchmarks for assessing visual commonsense in language models and compares multimodal and unimodal models, revealing limited gains from multimodal training.

Findings

Visual commonsense knowledge is similar in multimodal and unimodal models.

Multimodal training does not significantly improve visual commonsense knowledge.

Benchmarks effectively measure visual commonsense in language models.

Abstract

There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.