MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Training

TL;DR

MaD-Mix introduces a novel, efficient framework for creating multi-modal data mixtures in vision-language model training, reducing manual tuning and accelerating training while handling missing modalities.

Contribution

It formulates data mixing as modality-aware domain alignment maximization using closed-form scores, enabling scalable and automatic mixture design for VLMs.

Findings

Reduces training steps by 22% compared to human-tuned mixtures.

Accelerates training across diverse benchmarks with minimal overhead.

Improves accuracy in complex tri-modal scenarios.

Abstract

Vision-Language Models (VLMs) are typically trained on a diverse set of multi-modal domains, yet current practices rely on costly manual tuning. We propose MaD-Mix, a principled and computationally efficient framework that derives multi-modal data mixtures for VLM training. MaD-Mix formulates data mixing as modality-aware domain alignment maximization and obtains closed-form multi-modal alignment scores from the Fenchel dual through inter-modal coupling variables. MaD-Mix systematically handles domains with missing modalities, allowing for the integration of language-only domains. Empirical evaluations across 0.5B and 7B models demonstrate that MaD-Mix accelerates VLM training across diverse benchmarks. MaD-Mix matches human-tuned data mixtures using 22% fewer training steps in image-text instruction tuning. In complex tri-modal video-image-text scenarios, where manual tuning becomes impractical, MaD-Mix boosts average accuracy over uniform weights, with negligible mixture computation overhead (< 1 GPU-hour), enabling scalable mixture design for modern VLM pipelines.