RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness

TL;DR

RobustMerge is a training-free, parameter-efficient method for merging multi-task large language models, focusing on maintaining direction robustness through singular value compensation and cross-task normalization, leading to improved generalization.

Contribution

The paper introduces RobustMerge, a novel merging technique that preserves direction robustness without additional training, addressing limitations of existing full fine-tuning merging methods.

Findings

RobustMerge outperforms existing merging methods on diverse multimodal tasks.

It maintains direction robustness by compensating for singular value gaps.

The method enhances unseen task generalization and model stability.

Abstract

Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relation for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness. Code is available at https://github.com/AuroraZengfh/RobustMerge.