Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs

TL;DR

This paper introduces Locate-then-Merge, a parameter fusion method that preserves language skills while integrating visual capabilities in multimodal LLMs, reducing catastrophic forgetting without additional training.

Contribution

It proposes a training-free neuron-level parameter fusion framework that selectively merges important neurons to retain language ability and enhance visual adaptation in multimodal LLMs.

Findings

Outperforms existing model merging methods on 13 benchmarks

Effectively reduces context hallucination in generation

Preserves language skills while integrating visual capabilities

Abstract

Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM's language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts--neurons likely responsible for newly acquired visual capabilities--while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.