Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

TL;DR

This survey explores methods, applications, and future challenges of model merging in large language models, emphasizing theoretical foundations, algorithmic techniques, and practical uses like multi-task learning and federated systems.

Contribution

It provides a comprehensive taxonomy and systematic review of model merging techniques, theoretical insights, and application scenarios in the context of large language models.

Findings

Established theoretical foundations including loss landscape geometry and mode connectivity.

Reviewed diverse merging algorithms such as weight averaging, task vector arithmetic, and mixture-of-experts.

Identified open challenges and future research directions in model merging for LLMs.

Abstract

Model merging combines the parameters of multiple neural networks into a single model without additional training. As fine-tuned large language models (LLMs) proliferate, merging offers a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey examines model merging in the LLM era through the \textbf{FUSE} taxonomy, organized along \textbf{F}oundations, \textbf{U}nification Strategies, \textbf{S}cenarios, and \textbf{E}cosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry and mode connectivity, then systematically review the algorithmic space spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, and federated learning, and survey the supporting ecosystem of tools and evaluation benchmarks. Finally, we identify key open challenges and future directions, aiming to equip researchers and practitioners with a structured foundation for advancing model merging.