Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

TL;DR

This paper shows that simple fine-tuning adjustments can effectively prevent catastrophic forgetting in multimodal large language models, offering practical strategies for model adaptation and continual learning.

Contribution

It introduces straightforward fine-tuning techniques and a data-hybrid training strategy that mitigate forgetting and enhance continual learning in MLLMs.

Findings

Regularization prevents forgetting of out-of-distribution images.

Data-hybrid training addresses task-specific overfitting.

Appropriate fine-tuning improves continual learning performance.

Abstract

The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.