Activated Parameter Locating via Causal Intervention for Model Merging

TL;DR

This paper introduces a causal intervention-based method called Activated Parameter Locating (APL) for model merging, which improves parameter selection by estimating importance more accurately, leading to better conflict resolution and performance.

Contribution

The paper proposes a novel APL method utilizing causal intervention for precise parameter importance estimation in model merging, along with a gradient approximation strategy to reduce computational complexity.

Findings

APL outperforms existing methods in in-domain and out-of-domain settings.

The gradient approximation reduces computational costs without sacrificing accuracy.

Experiments demonstrate improved conflict mitigation and model performance.

Abstract

Model merging combines multiple homologous models into one model, achieving convincing generalization without the necessity of additional training. A key challenge in this problem is resolving parameter redundancies and conflicts across multiple models. Existing models have demonstrated that dropping a portion of delta parameters can alleviate conflicts while maintaining performance. However, these methods often drop parameters either randomly or based on magnitude, overlooking task-specific information embedded in fine-tuned models. In this paper, we propose an Activated Parameter Locating (APL) method that utilizes causal intervention to estimate parameter importance, enabling more precise parameter drops and better conflict mitigation. Moreover, to reduce the computational complexity associated with a large number of parameter partitions, we also introduce a theoretically supported gradient approximation strategy for APL. Experiments on model merging within both in-domain and out-of-domain settings, along with associated analyses, showcase the effectiveness of APL.