DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling

TL;DR

DELLA-Merging introduces a magnitude-based sampling technique for model merging that reduces interference and improves performance across multiple benchmarks by employing a novel pruning and rescaling method.

Contribution

The paper presents DELLA-Merging, a new model merging approach using MAGPRUNE, which outperforms existing methods by effectively reducing interference through magnitude-based parameter sampling.

Findings

DELLA-Merging improves performance by 2.4 points on average over baseline methods.

MAGPRUNE outperforms DARE and TIES in model merging tasks.

The method achieves an 11.1-point improvement over no-pruning baselines.

Abstract

With the proliferation of domain-specific models, model merging has emerged as a set of techniques that combine the capabilities of multiple models into one that can multitask without the cost of additional training. In this paper, we propose a new model merging technique, Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE, which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the parameters in order of their magnitude and assigns higher dropout probabilities (p) to parameters with lower ranks corresponding to lower magnitudes. To approximate the original embeddings, MAGPRUNE employs a rescaling operation on the parameters that survive the random dropping by 1/(1 - p). On three different expert models considered for merging (LM, Math, Code) and corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an average improvement of 2.4 points over baseline methods employing delta parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over DARE), and 11.1 points over the no-pruning baseline (TA). We release the source code at: https://github.com/declare-lab/della.