Navigating the Accuracy-Size Trade-Off with Flexible Model Merging

TL;DR

FlexMerge is a versatile, data-free framework for model merging that balances accuracy and size, revealing new insights into algorithm performance across different model sizes.

Contribution

The paper introduces FlexMerge, enabling flexible, multi-size model merging and systematic analysis of accuracy-size trade-offs across various algorithms.

Findings

Modestly larger merged models significantly improve accuracy (up to 13.5%).

Algorithm performance rankings vary with model size.

FlexMerge works effectively on vision and NLP benchmarks with up to 30 tasks.

Abstract

Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to the fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high storage costs. We propose FlexMerge, a novel data-free model merging framework that: (a) flexibly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all fine-tuned models; and (b) supports multiple merging algorithms in a unified framework. Using FlexMerge, we systematically characterize the accuracy-size trade-off of different algorithms. Our study reveals two key findings: first, even modestly larger merged models can yield steep accuracy gains (up to 13.5% when just doubling the size); second, algorithm rankings are not consistent as size increases, with some methods overtaking others beyond the one-model regime. These results uncover a new design dimension for model merging: developing and comparing algorithms across the full spectrum of sizes rather than only at the single-model limit. Extensive experiments on vision and NLP benchmarks, with up to 30 tasks, confirm the generality and practicality of FlexMerge.