CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

TL;DR

This paper introduces CoVFT, a framework that enhances multimodal large language models by incorporating context-aware visual fine-tuning, leading to improved stability and state-of-the-art results across benchmarks.

Contribution

The paper proposes a novel context-aware fine-tuning method with CVE and CoMoE modules to address visual preference conflicts in MLLMs, improving stability and performance.

Findings

CoVFT achieves state-of-the-art results on 12 benchmarks.

Fine-tuning with CoVFT outperforms larger models with frozen encoders.

Visual fine-tuning stability is improved by context-aware modules.

Abstract

Multimodal large language models (MLLMs) achieve remarkable progress in cross-modal perception and reasoning, yet a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen? Despite the success of models such as LLaVA and Qwen-VL, inconsistent design choices and heterogeneous training setups hinder a unified understanding of visual fine-tuning (VFT) in MLLMs. Through a configuration-aligned benchmark, we find that existing VFT methods fail to consistently outperform the frozen baseline across multimodal tasks. Our analysis suggests that this instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates under diverse multimodal context. To address this issue, we propose the Context-aware Visual Fine-tuning (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation. By integrating a Context Vector Extraction (CVE) and a Contextual Mixture-of-Experts (CoMoE) module, CoVFT decomposes conflicting optimization signals and enables stable, context-sensitive visual updates. Extensive experiments on 12 multimodal benchmarks demonstrate that CoVFT achieves state-of-the-art performance with superior stability. Notably, fine-tuning a 7B MLLM with CoVFT surpasses the average performance of its 13B counterpart, revealing substantial untapped potential in visual encoder optimization within MLLMs.