Hidden in plain sight: VLMs overlook their visual representations

TL;DR

This paper investigates why vision language models (VLMs) underperform on vision-centric tasks, revealing that they rely heavily on language priors and do not effectively utilize visual representations, highlighting key failure modes.

Contribution

The study provides a comprehensive analysis of VLMs' limitations in visual tasks, identifying the bottleneck as the language model's influence over visual information processing.

Findings

VLMs perform worse than visual encoders on vision tasks

Performance drops to near-chance levels on benchmarks like depth estimation

VLMs are heavily influenced by language priors, limiting visual understanding

Abstract

Language provides a natural interface to specify and evaluate performance on visual tasks. To realize this possibility, vision language models (VLMs) must successfully integrate visual and linguistic information. Our work compares VLMs to a direct readout of their visual encoders to understand their ability to integrate across these modalities. Across a series of vision-centric benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance. We investigate these results through a series of analyses across the entire VLM: namely 1) the degradation of vision representations, 2) brittleness to task prompt, and 3) the language model's role in solving the task. We find that the bottleneck in performing these vision-centric tasks lies in this third category; VLMs are not effectively using visual information easily accessible throughout the entire model, and they inherit the language priors present in the LLM. Our work helps diagnose the failure modes of open-source VLMs, and presents a series of evaluations useful for future investigations into visual understanding within VLMs.