Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models

TL;DR

This paper identifies visual representation degradation in multimodal large language models caused by training objectives and proposes a predictive regularization method to preserve visual fidelity, improving overall vision-language performance.

Contribution

It introduces Predictive Regularization (PRe) to maintain visual features in MLLMs, addressing internal visual degradation and enhancing multimodal understanding.

Findings

Mitigating visual degradation improves vision-language task performance.

Degradation occurs in middle layers due to training objectives.

PRe effectively preserves visual features in internal representations.

Abstract

While Multimodal Large Language Models (MLLMs) excel at vision-language tasks, the cost of their language-driven training on internal visual foundational competence remains unclear. In this paper, we conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs. Specifically, we find that compared to the initial visual features, the visual representation in the middle layers of LLM exhibits both a degradation in global function and patch structure. We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation. We argue that a robust MLLM requires both strong cross-modal reasoning and core visual competence, and propose Predictive Regularization (PRe) to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations. Extensive experiments confirm that mitigating this visual degradation effectively boosts vision-language performance, underscoring the critical importance of fostering robust internal visual representations within MLLMs for comprehensive multimodal understanding.