Competition and Attraction Improve Model Fusion

TL;DR

This paper introduces M2N2, an evolutionary algorithm for model merging that dynamically explores parameter combinations, preserves diversity through competition, and effectively evolves models from scratch, achieving state-of-the-art results.

Contribution

The paper presents M2N2, a novel evolutionary approach that enables dynamic, diverse, and efficient model merging, including from scratch, surpassing prior fixed-group methods.

Findings

M2N2 can evolve models from scratch with performance comparable to CMA-ES.

It scales to merge language and image models, achieving state-of-the-art results.

M2N2 preserves crucial model capabilities beyond explicit optimization.

Abstract

Model merging is a powerful technique for integrating the specialized knowledge of multiple machine learning models into a single model. However, existing methods require manually partitioning model parameters into fixed groups for merging, which restricts the exploration of potential combinations and limits performance. To overcome these limitations, we propose Model Merging of Natural Niches (M2N2), an evolutionary algorithm with three key features: (1) dynamic adjustment of merging boundaries to progressively explore a broader range of parameter combinations; (2) a diversity preservation mechanism inspired by the competition for resources in nature, to maintain a population of diverse, high-performing models that are particularly well-suited for merging; and (3) a heuristicbased attraction metric to identify the most promising pairs of models for fusion. Our experimental results demonstrate, for the first time, that model merging can be used to evolve models entirely from scratch. Specifically, we apply M2N2 to evolve MNIST classifiers from scratch and achieve performance comparable to CMA-ES, while being computationally more efficient. Furthermore, M2N2 scales to merge specialized language and image generation models, achieving state-of-the-art performance. Notably, it preserves crucial model capabilities beyond those explicitly optimized by the fitness function, highlighting its robustness and versatility. Our code is available at https://github.com/SakanaAI/natural_niches