Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success

TL;DR

This paper investigates the factors influencing the success of model merging, revealing that gradient alignment metrics are key indicators across various methods and architectures.

Contribution

It introduces an architecture-agnostic framework using interpretable metrics to predict and understand model mergeability, highlighting the importance of gradient alignment.

Findings

Gradient alignment metrics are the most fundamental signals of compatibility.

Success drivers vary across architectures and merging methods.

Certain methods like TIES have distinct mergeability 'fingerprints'.

Abstract

Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient L_2 distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find architecture- and method-specific variation in success drivers (64.0% average top-5 metric overlap; 79.3% sign agreement), with certain methods, notably TIES, exhibiting distinct ``fingerprints'' that diverge from the broader consensus. Crucially, however, gradient alignment metrics consistently emerge as the most fundamental signals of compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.