LLMs Can Compensate for Deficiencies in Visual Representations

TL;DR

This paper explores how language components in vision-language models can compensate for weak visual features, revealing a dynamic division of labor that enhances multimodal task performance.

Contribution

It demonstrates that language backbones in VLMs can offset limitations in visual representations, suggesting new architectural directions.

Findings

CLIP visual features contain accessible semantic information

Language decoders can compensate for visual deficiencies

Reduced visual contextualization can be mitigated by language models

Abstract

Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.