Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis

TL;DR

This paper introduces a simple, learning-free multimodal dataset distillation method using CLIP and unCLIP, which synthesizes data efficiently and generalizes well across different model architectures, outperforming existing methods.

Contribution

The proposed framework eliminates the need for large-scale training and joint optimization, enabling scalable, architecture-agnostic multimodal dataset distillation.

Findings

Outperforms optimization-based distillation methods

Achieves state-of-the-art cross-architecture generalization

Efficiently synthesizes multimodal data without training

Abstract

Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.