Will it Merge? On The Causes of Model Mergeability

TL;DR

This paper investigates the factors influencing the success of model merging, proposing a measurable definition of mergeability and highlighting the importance of base model knowledge in determining mergeability outcomes.

Contribution

It introduces a concrete definition of mergeability and demonstrates that models fine-tuned on familiar data are more mergeable, proposing a weighted merging technique to preserve weak knowledge.

Findings

Models fine-tuned on familiar data are more mergeable.

Base model knowledge significantly affects mergeability.

Weighted merging better preserves weak knowledge.

Abstract

Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model.