Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization

TL;DR

This paper introduces a scalable method that uses model merging as a proxy to efficiently estimate the performance of different data mixtures for multimodal models, reducing the need for costly training runs.

Contribution

It proposes a novel approach that leverages model merging to predict data mixture effectiveness, significantly simplifying the optimization process for multimodal model fine-tuning.

Findings

Merged proxy models correlate highly with actual data mixture models

The method reduces computational costs for data mixture optimization

Extensive experiments on 14 benchmarks validate the approach

Abstract

Selecting the best data mixture is critical for successful Supervised Fine-Tuning (SFT) of Multimodal Large Language Models. However, determining the optimal mixture weights across multiple domain-specific datasets remains a significant bottleneck due to the combinatorial search space and the high cost associated with even a single training run. This is the so-called Data Mixture Optimization (DMO) problem. On the other hand, model merging unifies domain-specific experts through parameter interpolation. This strategy is efficient, as it only requires a single training run per domain, yet oftentimes leads to suboptimal models. In this work, we take the best of both worlds, studying model merging as an efficient strategy for estimating the performance of different data mixtures. We train domain-specific multimodal experts and evaluate their weighted parameter-space combinations to estimate the efficacy of corresponding data mixtures. We conduct extensive experiments on 14 multimodal benchmarks, and empirically demonstrate that the merged proxy models exhibit a high rank correlation with models trained on actual data mixtures. This decouples the search for optimal mixtures from the resource-intensive training process, thereby providing a scalable and efficient strategy for navigating the complex landscape of mixture weights. Code is publicly available at https://github.com/BerasiDavide/mLLMs_merging_4_DMO.