AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse

TL;DR

AutoMerge introduces a systematic, search-based framework for model merging that effectively handles diverse architectures and domains, overcoming limitations of existing techniques and enhancing model reuse.

Contribution

This paper presents AutoMerge, the first search-based model merging framework that adapts to heterogeneous models across domains, improving upon existing methods' limitations.

Findings

Existing merging techniques are inconsistent across models and domains.

A single merging method cannot effectively handle heterogeneous model structures.

Hyperparameter sensitivity limits the broader applicability of current merging techniques.

Abstract

Software reuse has long been recognized as a critical and widely studied topic in software engineering, offering substantial benefits in reducing development costs, improving software quality, and enhancing operational efficiency. This paradigm extends into deep learning through model reuse. Recently, model merging has emerged in the domain of large language models (LLMs) as a training-free approach that takes multiple task-specific models with the same architecture as source models and merges them without retraining, enhancing model reuse within LLMs. However, no prior work has systematically investigated whether such an approach can be effectively applied to other deep learning models with different architectures across domains. To bridge this gap, we present the first systematic study that evaluates five model merging techniques on three distinct model architectures across three domains: LLMs, image classification, and autonomous driving. Our findings reveal that directly applying existing model merging techniques leads to highly inconsistent results and falls notably short of their success within LLMs. Moreover, a single model merging technique often fails to handle the heterogeneous structural properties within a model, limiting its applicability to different model architectures across domains. Furthermore, the effectiveness of model merging techniques is highly sensitive to hyperparameter configurations, thereby constraining their potential for broader adoption. Inspired by these insights, we propose AutoMerge, a novel search-based model merging framework that first segments complex models into multiple heterogeneous blocks and then systematically explores the merging space to identify the merging technique and its hyperparameter configuration.