Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

TL;DR

Model-Dowser is a novel sparse fine-tuning method that preserves important parameters to prevent catastrophic forgetting in multimodal large language models, improving performance and scalability.

Contribution

It introduces a principled importance scoring mechanism for parameters, enabling effective, resource-efficient mitigation of forgetting during fine-tuning.

Findings

Outperforms prior methods in mitigating catastrophic forgetting

Effective on large-scale multimodal models like LLaVA and NVILA

Remains resource-efficient and scalable to multi-billion-parameter models

Abstract

Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. However, such adaptation often leads to a degradation in generalization on pretrained tasks, a phenomenon known as Catastrophic Forgetting. Existing methods that aim to mitigate this issue either become ineffective when fine-tuning deeper layers of the language decoder or scale poorly with increasing model size. To address these limitations, we propose Model-Dowser, a novel sparse fine-tuning approach for MLLMs. Model-Dowser measures a principled importance score for each model parameter with respect to pretrained generalization (prior to downstream adaptation) by jointly considering weight magnitudes, input activations, and output sensitivities. During fine-tuning, Model-Dowser selectively preserves high-importance parameters and updates the remaining. Comprehensive experiments on two representative MLLMs, LLaVA and NVILA, demonstrate that Model-Dowser effectively mitigates catastrophic forgetting and consistently outperforms prior methods, while remaining resource-efficient and scalable to multi-billion-parameter models.