Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation

TL;DR

This paper introduces RepBlend, a novel framework for multimodal dataset distillation that alleviates modality collapse by blending representations and balancing supervision, leading to improved cross-modal learning and efficiency.

Contribution

RepBlend is the first method to address modality collapse in MDD by representation blending and symmetric projection matching, enhancing intra-modal diversity and cross-modal alignment.

Findings

Outperforms prior MDD methods on Flickr-30K and MS-COCO

Achieves up to 9.4 IR@10 and 6.3 TR@10 improvements

Provides up to 6.7× speedup in distillation

Abstract

Multimodal Dataset Distillation (MDD) seeks to condense large-scale image-text datasets into compact surrogates while retaining their effectiveness for cross-modal learning. Despite recent progress, existing MDD approaches often suffer from \textit{\textbf{Modality Collapse}}, characterized by over-concentrated intra-modal representations and enlarged distributional gap across modalities. In this paper, at the first time, we identify this issue as stemming from a fundamental conflict between the over-compression behavior inherent in dataset distillation and the cross-modal supervision imposed by contrastive objectives. To alleviate modality collapse, we introduce \textbf{RepBlend}, a novel MDD framework that weakens overdominant cross-modal supervision via representation blending, thereby significantly enhancing intra-modal diversity. Additionally, we observe that current MDD methods impose asymmetric supervision across modalities, resulting in biased optimization. To address this, we propose symmetric projection trajectory matching, which synchronizes the optimization dynamics using modality-specific projection heads, thereby promoting balanced supervision and enhancing cross-modal alignment. Experiments on Flickr-30K and MS-COCO show that RepBlend consistently outperforms prior state-of-the-art MDD methods, achieving significant gains in retrieval performance (e.g., +9.4 IR@10, +6.3 TR@10 under the 100-pair setting) and offering up to 6.7$\times$ distillation speedup.