Explore How to Inject Beneficial Noise in MLLMs

TL;DR

This paper introduces a novel fine-tuning method for Multimodal Large Language Models that injects beneficial noise to improve cross-modal alignment and performance, surpassing traditional fine-tuning techniques with minimal additional parameters.

Contribution

The paper proposes MuNG, a multimodal noise generator that dynamically analyzes cross-modal relationships to inject task-adaptive noise, enhancing MLLMs without full fine-tuning.

Findings

Outperforms full fine-tuning and existing methods

Requires only 1-2% additional parameters

Improves cross-modal representation and downstream task performance

Abstract

Multimodal Large Language Models (MLLMs) have played an increasingly important role in multimodal intelligence. However, the existing fine-tuning methods often ignore cross-modal heterogeneity, limiting their full potential. In this work, we propose a novel fine-tuning strategy by injecting beneficial random noise, which outperforms previous methods and even surpasses full fine-tuning, with minimal additional parameters. The proposed Multimodal Noise Generator (MuNG) enables efficient modality fine-tuning by injecting customized noise into the frozen MLLMs. Specifically, we reformulate the reasoning process of MLLMs from a variational inference perspective, upon which we design a multimodal noise generator that dynamically analyzes cross-modal relationships in image-text pairs to generate task-adaptive beneficial noise. Injecting this type of noise into the MLLMs effectively suppresses irrelevant semantic components, leading to significantly improved cross-modal representation alignment and enhanced performance on downstream tasks. Experiments on two mainstream MLLMs, QwenVL and LLaVA, demonstrate that our method surpasses full-parameter fine-tuning and other existing fine-tuning approaches, while requiring adjustments to only about $1\sim2\%$ additional parameters. The relevant code is uploaded in the supplementary.