Multi-Modal Dataset Distillation in the Wild

TL;DR

This paper introduces MDW, a novel framework for distilling noisy multi-modal datasets into compact, clean datasets, improving training efficiency and robustness in multi-modal models with significant scalability and noise tolerance.

Contribution

MDW is the first framework to effectively distill noisy multi-modal datasets into clean, compact datasets using learnable correspondences and dual-track collaborative learning.

Findings

MDW surpasses prior methods by over 15% across various compression ratios.

MDW effectively handles noisy web-crawled multi-modal data.

Distilled datasets improve model training efficiency and robustness.

Abstract

Recent multi-modal models have shown remarkable versatility in real-world applications. However, their rapid development encounters two critical data challenges. First, the training process requires large-scale datasets, leading to substantial storage and computational costs. Second, these data are typically web-crawled with inevitable noise, i.e., partially mismatched pairs, severely degrading model performance. To these ends, we propose Multi-modal dataset Distillation in the Wild, i.e., MDW, the first framework to distill noisy multi-modal datasets into compact clean ones for effective and efficient model training. Specifically, MDW introduces learnable fine-grained correspondences during distillation and adaptively optimizes distilled data to emphasize correspondence-discriminative regions, thereby enhancing distilled data's information density and efficacy. Moreover, to capture robust cross-modal correspondence prior knowledge from real data, MDW proposes dual-track collaborative learning to avoid the risky data noise, alleviating information loss with certifiable noise tolerance. Extensive experiments validate MDW's theoretical and empirical efficacy with remarkable scalability, surpassing prior methods by over 15% across various compression ratios, highlighting its appealing practicality for applications with diverse efficacy and resource needs.