Why do LLaVA Vision-Language Models Reply to Images in English?

TL;DR

This paper reveals a multilingual bias in LLaVA vision-language models where images increase the likelihood of English responses, and investigates the underlying causes through ablation and mechanistic analysis.

Contribution

It identifies the source of the bias in the language modeling component and shows that replacing the language backbone reduces the bias, providing insights for more inclusive VLMs.

Findings

Visual inputs are not mapped to the same space as text inputs.

Switching to a bilingual language model reduces the bias.

Intervening on attention layers can mitigate the bias.

Abstract

We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models' internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modelling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts.