Exploring Model Kinship for Merging Large Language Models

TL;DR

This paper introduces the concept of model kinship to understand and improve the process of merging large language models, demonstrating that kinship-guided merging enhances performance and mitigates local optima issues.

Contribution

It proposes the novel concept of model kinship, analyzes its relation to merging gains, and develops a kinship-guided merging strategy to improve LLM evolution.

Findings

Model kinship correlates with performance gains in merging.

Kinship-guided merging improves benchmark results.

The approach mitigates local optima during model evolution.

Abstract

Model merging has emerged as a key technique for enhancing the capabilities and efficiency of Large Language Models (LLMs). The open-source community has driven model evolution by iteratively merging existing models, yet a principled understanding of the gains and underlying factors in model merging remains limited. In this work, we study model evolution through iterative merging, drawing an analogy to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. Through comprehensive empirical analysis, we show that model kinship is closely linked to the performance improvements achieved by merging, providing a useful criterion for selecting candidate models. Building on this insight, we propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can improve benchmark performance. Specifically, we discover that incorporating model kinship as a guiding criterion enables continuous merging while mitigating performance degradation caused by local optima, thereby facilitating more effective model evolution. Code is available at https://github.com/zjunlp/ModelKinship.