Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

TL;DR

This study investigates how vision-and-language training influences language models, finding it enhances task deployment of taxonomic knowledge without fundamentally altering the underlying representations of this knowledge.

Contribution

The paper demonstrates that VL training improves task-specific application of taxonomic knowledge without significantly changing its core representations.

Findings

VL models outperform text-only models on taxonomic question-answering tasks

Taxonomic knowledge remains largely unchanged by VL training

VL training affects how models represent taxonomic vs. non-taxonomic concepts

Abstract

Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.