ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation

TL;DR

ImageBindDC introduces a novel data condensation method for multimodal data that preserves inter-modal dependencies using a characteristic function loss, enabling efficient training with significantly fewer data points.

Contribution

The paper proposes a new multimodal data condensation framework using a characteristic function loss in the Fourier domain to better preserve inter-modal relationships.

Findings

Achieves lossless performance with only 5 condensed points per class on NYU-v2.

Outperforms previous methods with an 8.2% accuracy improvement.

Reduces condensation time by more than 4 times.

Abstract

Data condensation techniques aim to synthesize a compact dataset from a larger one to enable efficient model training, yet while successful in unimodal settings, they often fail in multimodal scenarios where preserving intricate inter-modal dependencies is crucial. To address this, we introduce ImageBindDC, a novel data condensation framework operating within the unified feature space of ImageBind. Our approach moves beyond conventional distribution-matching by employing a powerful Characteristic Function (CF) loss, which operates in the Fourier domain to facilitate a more precise statistical alignment via exact infinite moment matching. We design our objective to enforce three critical levels of distributional consistency: (i) uni-modal alignment, which matches the statistical properties of synthetic and real data within each modality; (ii) cross-modal alignment, which preserves pairwise semantics by matching the distributions of hybrid real-synthetic data pairs; and (iii) joint-modal alignment, which captures the complete multivariate data structure by aligning the joint distribution of real data pairs with their synthetic counterparts. Extensive experiments highlight the effectiveness of ImageBindDC: on the NYU-v2 dataset, a model trained on just 5 condensed datapoints per class achieves lossless performance comparable to one trained on the full dataset, achieving a new state-of-the-art with an 8.2\% absolute improvement over the previous best method and more than 4$\times$ less condensation time.