Training-free Heterogeneous Model Merging

TL;DR

This paper introduces a novel training-free framework for merging heterogeneous models with different architectures, using layer alignment and elastic neuron zipping, enabling effective model reuse across vision and NLP tasks.

Contribution

It proposes innovative methods for merging models with differing depths and widths without retraining, expanding the applicability of model merging techniques.

Findings

Heterogeneous model merging achieves comparable performance to homogeneous merging.

Layer alignment effectively handles depth discrepancies.

Elastic neuron zipping manages width heterogeneity without performance loss.

Abstract

Model merging has attracted significant attention as a powerful paradigm for model reuse, facilitating the integration of task-specific models into a singular, versatile framework endowed with multifarious capabilities. Previous studies, predominantly utilizing methods such as Weight Average (WA), have shown that model merging can effectively leverage pretrained models without the need for laborious retraining. However, the inherent heterogeneity among models poses a substantial constraint on its applicability, particularly when confronted with discrepancies in model architectures. To overcome this challenge, we propose an innovative model merging framework designed for heterogeneous models, encompassing both depth and width heterogeneity. To address depth heterogeneity, we introduce a layer alignment strategy that harmonizes model layers by segmenting deeper models, treating consecutive layers with similar representations as a cohesive segment, thus enabling the seamless merging of models with differing layer depths. For width heterogeneity, we propose a novel elastic neuron zipping algorithm that projects the weights from models of varying widths onto a common dimensional space, eliminating the need for identical widths. Extensive experiments validate the efficacy of these proposed methods, demonstrating that the merging of structurally heterogeneous models can achieve performance levels comparable to those of homogeneous merging, across both vision and NLP tasks. Our code is publicly available at https://github.com/zju-vipa/training_free_heterogeneous_model_merging.