Same Answer, Different Representations: Hidden instability in VLMs

TL;DR

This paper introduces a new evaluation framework for Vision Language Models that assesses internal representation stability and reveals that larger models are not necessarily more robust, with internal drift often occurring despite stable outputs.

Contribution

The work presents a representation-aware and frequency-aware evaluation method for VLMs, uncovering internal instability and failure modes not captured by output-level metrics.

Findings

Models often show internal representation drift despite stable predictions.

Larger models do not necessarily have improved robustness; they can be more sensitive.

Perturbations impact reasoning and hallucination differently, sometimes reducing false positives.

Abstract

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.