System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

TL;DR

This paper investigates how system-mediated attention imbalances in vision-language models contribute to hallucinations like the yes-bias, and demonstrates that redistributing attention can effectively mitigate this issue.

Contribution

It introduces a holistic system-mediated framework for understanding attention imbalances and shows how adjusting attention distribution reduces hallucinations in VLMs.

Findings

Redistributing attention from system to input modalities suppresses yes-bias.

System attention imbalances lead to reliance on coarse representations, causing hallucinations.

Adjusting attention improves model responses beyond existing mitigation methods.

Abstract

Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric interpretation of these imbalances, often prioritising increased image attention while giving less consideration to the roles of the other modalities. In this study, we evaluate a more holistic, system-mediated account, which attributes these imbalances to functionally redundant system weights that reduce attention to image and textual inputs. We show that this framework offers a useful empirical perspective on the yes-bias, a common form of hallucination in which VLMs indiscriminately respond `yes'. Causally redistributing attention from the system modality to image and textual inputs substantially suppresses this bias, often outperforming existing approaches. We further present evidence suggesting that system-mediated attention imbalances contribute to the yes-bias by encouraging a default reliance on coarse input representations, which are effective for some tasks but ill-suited to others. Taken together, these findings firmly establish system attention as a key factor in VLM hallucination and highlight its potential as a lever for mitigation.