Analyzing the Roles of Language and Vision in Learning from Limited Data

TL;DR

This paper investigates how language and vision contribute to learning in AI models, revealing that language alone can recover much of a vision-language model's performance by providing prior knowledge and reasoning capabilities.

Contribution

The study systematically ablates components of vision-language models to identify the specific roles of language and vision in learning from limited data.

Findings

Language models can recover most of VLM performance without visual input.

Language provides access to prior knowledge and reasoning.

Vision is less critical than language for certain learning tasks.

Abstract

Does language help make sense of the visual world? How important is it to actually see the world rather than having it described with words? These basic questions about the nature of intelligence have been difficult to answer because we only had one example of an intelligent system -- humans -- and limited access to cases that isolated language or vision. However, the development of sophisticated Vision-Language Models (VLMs) by artificial intelligence researchers offers us new opportunities to explore the contributions that language and vision make to learning about the world. We ablate components from the cognitive architecture of these models to identify their contributions to learning new tasks from limited data. We find that a language model leveraging all components recovers a majority of a VLM's performance, despite its lack of visual input, and that language seems to allow this by providing access to prior knowledge and reasoning.