Model Fusion via Neuron Interpolation
Phoomraphee Luenam, Andreas Spanopoulos, Amit Sant, Thomas Hofmann, Sotiris Anagnostidis, Sidak Pal Singh
TL;DR
This paper introduces neuron-centric model fusion algorithms that effectively combine multiple neural networks into a single model, outperforming previous methods especially in zero-shot and non-IID scenarios.
Contribution
The paper presents a novel family of neuron-based model fusion algorithms that incorporate neuron attribution scores and generalize across layer types, improving fusion performance.
Findings
Outperforms previous fusion techniques on benchmark datasets.
Effective in zero-shot and non-IID data scenarios.
Generalizes to arbitrary layer types.
Abstract
Model fusion aims to combine the knowledge of multiple models by creating one representative model that captures the strengths of all of its parents. However, this process is non-trivial due to differences in internal representations, which can stem from permutation invariance, random initialization, or differently distributed training data. We present a novel, neuron-centric family of model fusion algorithms designed to integrate multiple trained neural networks into a single network effectively regardless of training data distribution. Our algorithms group intermediate neurons of parent models to create target representations that the fused model approximates with its corresponding sub-network. Unlike prior approaches, our approach incorporates neuron attribution scores into the fusion process. Furthermore, our algorithms can generalize to arbitrary layer types. Experimental results…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
- The proposed framework is intuitive, flexible, and moves beyond simple weight permutation (like OTFusion or Git Re-Basin ) by actively fitting a new sub-network to an intermediate target representation - The incorporation of neuron attribution scores into the fusion objective is a novel contribution, theoretically allowing the process to prioritize salient features - The empirical results are strong, especially in zero-shot and non-IID "sharded" scenarios (Table 2, 15), where the method (KF Gr
- The gradient-based variants, which produce the best results, are admittedly sensitive to hyperparameters. The paper provides two starkly different configurations (Setting 1 vs. Setting 2, Table 9) without a clear ablation or principle for choosing between them. This significantly undermines the method's robustness and practicality - The central novelty claim—incorporating attribution scores —is weakly supported by the main experimental results. For the flagship ViT experiments (Table 15, 16),
- The decomposition of the fusion objective into grouping and approximation stages is a useful formalization. It reframes neuron matching as a clustering-and-refitting process, connecting prior permutation and alignment-based methods to a broader optimization viewpoint. - Integrating saliency measures to inform neuron grouping and weighting adds a novel, biologically-inspired dimension to the fusion literature, which has largely focused on structural similarity rather than neuron importance. - T
- **Title clarity**: Since “interpolation” is a standard aggregation term in the model merging literature, consider renaming to reflect the neuron-centric mechanism or saliency usage. - While the neuron clustering view is fresh, the method seems to **heavily build on Git-Rebasin’s activation matching** and other alignment-based model fusion works (e.g., _Ainsworth et al., Git Re-basin: Merging Models Effectively, 2023_). The contribution mainly lies in incorporating saliency and centroid fitting
- **Problem Motivation:** The paper correctly identifies a significant and practical problem: existing fusion methods struggle in zero-shot and non-IID settings, which are common in real-world applications like Federated Learning. - **Flexible Framework:** The proposed two-stage (grouping, fitting) framework is flexible. The K-means Fusion (KF) variant can naturally handle fusing models of different widths (as shown in Table 1, if it were filled out), and the gradient-based variant can be appli
- **High Complexity and Sensitivity:** The authors' best method (gradient-based KF) is admitted to be "sensitive to hyperparameters" and its lack of robustness is shown by requiring two different settings for the paper's own experiments. It is also computationally expensive, running 16-23x slower than baselines (Table 10). - **Overstated Saliency Contribution:** The claim of being the "first" to use saliency scores is factually incorrect, as the authors admit in Appendix C that prior work (Singh
1. It casts fusion as a principled representation matching problem, yielding a two-stage algorithm, which measure grouping error and approximation error. It decouples the traditional objective by introducing an auxiliary vector, enabling a more tractable decomposition of the cost function. 2. It incorporates neuron saliency into alignment, improving performance across our methods and enhancing existing approaches. 3. It provides a flexible opensource re-implementation of existing algorithms.
1. This method view DNN as a function parameterized by weights and for many model architectures, this function can be decomposed into many subfunctions. However, the rapid development of network architecture has made many SOTA network architectures no longer cascading, but containing many intricate connections. The authors do not discuss these complex network architectures. 2. Does a collection of pretrained base models to share the same network architecture, that is, do they need to be siamese
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Domain Adaptation and Few-Shot Learning · Explainable Artificial Intelligence (XAI)