Improving the Efficiency of Visually Augmented Language Models

TL;DR

This paper introduces BLIND-VALM, a more efficient visually-augmented language model that uses visually-grounded text representations instead of explicit images, achieving comparable or better performance with less complexity.

Contribution

The paper demonstrates that visually-grounded text representations can replace explicit images in visually-augmented language models, improving efficiency and scalability.

Findings

BLIND-VALM matches VALM's performance on key tasks.

Scaling BLIND-VALM improves results beyond VALM.

Using visually-grounded text simplifies the model architecture.

Abstract

Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.