Rethinking Weight-Averaged Model-merging

TL;DR

This paper reinterprets weight-averaged model merging through interpretability, analyzing weight structures, comparing in weight and feature spaces, and studying parameter scaling to understand its effectiveness and robustness.

Contribution

It provides a new interpretability perspective on weight averaging, revealing underlying mechanisms and conditions for effective model merging.

Findings

Weights encode structured representations aiding compatibility

Averaging in weight vs. feature space varies by architecture and dataset

Parameter scaling influences prediction stability and robustness

Abstract

Model merging, particularly through weight averaging, has shown surprising effectiveness in saving computations and improving model performance without any additional training. However, the interpretability of why and how this technique works remains unclear. In this work, we reinterpret weight-averaged model merging through the lens of interpretability and provide empirical insights into the underlying mechanisms that govern its behavior. We approach the problem from three perspectives: (1) we analyze the learned weight structures and demonstrate that model weights encode structured representations that help explain the compatibility of weight averaging; (2) we compare averaging in weight space and feature space across diverse model architectures (CNNs and ViTs) and datasets, aiming to expose under which circumstances what combination paradigm will work more effectively; (3) we study the effect of parameter scaling on prediction stability, highlighting how weight averaging acts as a form of regularization that contributes to robustness. By framing these analyses in an interpretability context, our work contributes to a more transparent and systematic understanding of model merging for stakeholders interested in the safety and reliability of untrained model combination methods. The code is available at https://github.com/billhhh/Rethink-Merge.