FREE-Merging: Fourier Transform for Efficient Model Merging

TL;DR

This paper introduces FREE-Merging, a novel model merging technique that filters harmful frequency domain interference to efficiently combine models, balancing performance, cost, and flexibility across multiple domains.

Contribution

It reveals frequency domain task interference in model merging and proposes FR-Merging and FREE-Merging to mitigate this interference with minimal overhead.

Findings

Effective filtering of frequency domain interference improves merging quality.

FREE-Merging balances performance, cost, and flexibility across tasks.

Demonstrated success across CV, NLP, and Multi-Modal domains.

Abstract

With the rapid growth of deep learning, there is an increasing availability of open-source models for various tasks. However, single fine-tuned models often fall short of meeting the diverse needs of users. Model merging has thus emerged as an efficient method to integrate the capabilities of existing models into a unified model. Nevertheless, existing model merging methods face challenging trade-offs between performance and deployment costs, primarily due to task interference. For the first time, we reveal that task interference is evident in the frequency domain of model parameters, yet current efforts only focus on spatial domain solutions, which are largely ineffective in addressing frequency domain interference. To mitigate the impact of frequency domain interference, we propose FR-Merging, an innovative method that effectively filters harmful frequency domain interference on the backbone with minimal computational overhead. Since performance loss is inevitable with cost-free methods, we propose a lightweight task-specific expert module that dynamically compensates for information loss during merging. This proposed framework, FREE-Merging (FR-Merging with experts), strikes a balanced trade-off between training cost, inference latency, storage requirements, and performance. We demonstrate the effectiveness of both FR-Merging and FREE-Merging on multiple tasks across CV, NLP, and Multi-Modal domains and show that they can be flexibly adapted to specific needs.