Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models

TL;DR

This paper proposes a novel fine-tuning approach for vision-language models called Mask Fine-Tuning (MFT), which reorganizes internal subnetworks via learnable gating, outperforming traditional weight update methods like LoRA and full fine-tuning.

Contribution

It introduces MFT for VLMs, demonstrating that reconfiguring internal structures can surpass weight update methods without changing the pre-trained backbone.

Findings

MFT outperforms LoRA variants in experiments.

MFT surpasses full fine-tuning in performance.

Effective adaptation from internal reorganization alone.

Abstract

Explorations in fine-tuning Vision-Language Models (VLMs), such as Low-Rank Adaptation (LoRA) from Parameter Efficient Fine-Tuning (PEFT), have made impressive progress. However, most approaches rely on explicit weight updates, overlooking the extensive representational structures already encoded in pre-trained models that remain underutilized. Recent works have demonstrated that Mask Fine-Tuning (MFT) can be a powerful and efficient post-training paradigm for language models. Instead of updating weights, MFT assigns learnable gating scores to each weight, allowing the model to reorganize its internal subnetworks for downstream task adaptation. In this paper, we rethink fine-tuning for VLMs from a structural reparameterization perspective grounded in MFT. We apply MFT to the language and projector components of VLMs with different language backbones and compare against strong PEFT baselines. Experiments show that MFT consistently surpasses LoRA variants and even full fine-tuning, achieving high performance without altering the frozen backbone. Our findings reveal that effective adaptation can emerge not only from updating weights but also from reestablishing connections among the model's existing knowledge. Code available at: https://github.com/Ming-K9/MFT-VLM