Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models

TL;DR

This paper introduces LaVer, a training framework that improves multimodal large language models by enhancing their visual representation learning through masked image modeling in latent space, addressing modality imbalance issues.

Contribution

The paper proposes LaVer, a novel training method that enables MLLMs to learn more discriminative visual features via latent space masked image modeling, improving visual understanding.

Findings

Enhanced visual attention allocation in MLLMs.

Superior performance on dense visual tasks.

Effective mitigation of modality imbalance.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in multimodal tasks. Despite their impressive performance, MLLMs suffer from the modality imbalance issue, where visual information is often underutilized compared to textual representations in deeper layers, leading to degraded visual performance or hallucinations. This issue stems from the predominant reliance on next-text-token-prediction during training, which fails to provide direct visual supervisory signals, resulting in progressive homogenization of visual representations throughout the layers. To this end, we propose Latent Visual Reconstruction (LaVer), a novel training framework that facilitates MLLMs in learning more discriminative visual representations via masked image modeling in the joint latent semantic space of LLM. Our method offers direct visual activation to MLLMs, which exhibit increased visual attention allocation, indicating enhanced utilization of visual information. Extensive experiments across diverse benchmarks prove the superiority of our approach in various scenarios, especially those requiring dense visual capabilities. Code of LaVer is available at https://github.com/Fir-lat/LaVer.