Investigating Spatial Attention Bias in Vision-Language Models

TL;DR

This paper uncovers a consistent left-to-right spatial attention bias in vision-language models, showing it persists across architectures and is not due to language reading direction or explicit training instructions, highlighting fundamental processing limitations.

Contribution

The study systematically identifies and characterizes a persistent spatial attention bias in VLMs, independent of language direction and training data annotations.

Findings

Models describe left content first in 97% of cases

Bias persists across different architectures

Training data annotations do not specify left-first ordering

Abstract

Vision-Language Models have demonstrated remarkable capabilities in understanding visual content, yet systematic biases in their spatial processing remain largely unexplored. This work identifies and characterizes a systematic spatial attention bias where VLMs consistently prioritize describing left-positioned content before right-positioned content in horizontally concatenated images. Through controlled experiments on image pairs using both open-source and closed-source models, we demonstrate that this bias persists across different architectures, with models describing left-positioned content first in approximately 97% of cases under neutral prompting conditions. Testing on an Arabic-finetuned model reveals that the bias persists despite right-to-left language training, ruling out language reading direction as the primary cause. Investigation of training dataset annotation guidelines from PixMo and Visual Genome reveals no explicit left-first ordering instructions, suggesting the bias is consistent with architectural factors rather than explicit training data instructions. These findings reveal fundamental limitations in how current VLMs process spatial information.