Model Merging: Foundations and Algorithms

TL;DR

This thesis explores model merging as an alternative to model replacement in deep learning, proposing algorithms and theoretical insights for combining neural networks directly in weight space without access to training data.

Contribution

It introduces novel algorithms and theoretical frameworks for model merging in single-task and multi-task settings, including C$^2$M$^3$, TSV-Merge, MASS, and MERGE$^3$, advancing the understanding and efficiency of model composition.

Findings

C$^2$M$^3$ aligns models into a shared parameter space for meaningful weight averaging.

Task vectors exhibit low-rank gradient structures enabling effective compression.

MERGE$^3$ reduces evaluation costs by up to 50 times while maintaining solution quality.

Abstract

Modern deep learning usually treats models as separate artifacts: trained independently, specialized for particular purposes, and replaced when improved versions appear. This thesis studies model merging as an alternative paradigm: combining independently trained neural networks directly in weight space, with little or no optimization and without requiring access to the original training data. The thesis considers two main regimes. In the single-task setting, where models share an objective but differ in initialization, we introduce C$^2$M$^3$, a cycle-consistent merging algorithm based on Frank-Wolfe optimization. C$^2$M$^3$ aligns multiple networks into a shared, reference-free parameter space, making weight averaging meaningful without privileging any individual model. In the multi-task setting, where models are fine-tuned for different downstream tasks from a common pretrained initialization, we first develop a theoretical account of task vectors as approximate gradients. This explains both the effectiveness and the limitations of task arithmetic. Building on this view, we show that task vectors inherit the low-rank structure of gradients and introduce Task Singular Vectors (TSV), a decomposition that enables compression and interference reduction through TSV-Merge. We then present MASS, an input-adaptive routing method that uses TSV geometry to select task-relevant subspaces at inference time. Finally, we introduce MERGE$^3$, an evolutionary merging framework that uses Item Response Theory to reduce evaluation costs by up to 50$\times$ while preserving solution quality. Together, these contributions provide theoretical and algorithmic foundations for model merging, supporting a paradigm in which learned capabilities can be composed, reused, and extended across models.