Omnimodal Dataset Distillation via High-order Proxy Alignment

TL;DR

This paper introduces HoPA, a novel method for omnimodal dataset distillation that captures high-order cross-modal alignments, enabling scalable and effective compression across multiple heterogeneous modalities.

Contribution

It proposes a unified high-order proxy alignment approach that overcomes complexity issues in omnimodal dataset distillation, with theoretical and empirical validation.

Findings

HoPA achieves superior compression-performance trade-offs.

Theoretical analysis supports the method's rationality.

Extensive experiments demonstrate effectiveness across benchmarks.

Abstract

Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.