MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

TL;DR

This paper introduces advanced techniques for merging expert models in Mixture-of-Experts frameworks, addressing challenges with model divergence and architecture differences to improve performance and reduce fine-tuning.

Contribution

The paper presents novel MoE merging methods that handle heterogeneous architectures and mitigate parameter interference, surpassing existing simple averaging approaches.

Findings

Effective merging reduces fine-tuning requirements

Improved performance over state-of-the-art methods

Applicable to both homogeneous and heterogeneous models

Abstract

The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging.