When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

TL;DR

This paper identifies geometric over-alignment as a key cause of hallucinations in vision-language models and proposes bias-removal methods that improve factual accuracy without extra training.

Contribution

It provides the first quantitative analysis of over-alignment in VLMs and introduces two bias mitigation techniques that enhance model reliability.

Findings

Significant reduction in hallucinations across multiple benchmarks.

Bias concentrates in top principal components of text subspace.

Training-free method improves factual accuracy without additional computation.

Abstract

Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.