The in-context inductive biases of vision-language models differ across modalities

TL;DR

This paper investigates how vision-language models exhibit different inductive biases depending on whether stimuli are presented visually or in text, revealing modality-specific generalization patterns and their implications.

Contribution

It introduces experimental paradigms to compare how vision-language models generalize across modalities, highlighting differences in shape bias and the influence of textual descriptions.

Findings

Models show a shape bias, especially with visual stimuli.

Text presentation affects generalization based on adjective order.

Bias effects vary across models and experimental paradigms.

Abstract

Inductive biases are what allow learners to make guesses in the absence of conclusive evidence. These biases have often been studied in cognitive science using concepts or categories -- e.g. by testing how humans generalize a new category from a few examples that leave the category boundary ambiguous. We use these approaches to study generalization in foundation models during in-context learning. Modern foundation models can condition on both vision and text, and differences in how they interpret and learn from these different modalities is an emerging area of study. Here, we study how their generalizations vary by the modality in which stimuli are presented, and the way the stimuli are described in text. We study these biases with three different experimental paradigms, across three different vision-language models. We find that the models generally show some bias towards generalizing according to shape over color. This shape bias tends to be amplified when the examples are presented visually. By contrast, when examples are presented in text, the ordering of adjectives affects generalization. However, the extent of these effects vary across models and paradigms. These results help to reveal how vision-language models represent different types of inputs in context, and may have practical implications for the use of vision-language models.