Multimodal Dataset Distillation via Phased Teacher Models

TL;DR

This paper introduces PTM-ST, a phased distillation framework that improves multimodal dataset distillation by modeling teacher learning dynamics across training phases, leading to better performance and stability.

Contribution

The paper proposes a novel phased teacher model with shortcut trajectory to better capture teacher dynamics and enhance dataset distillation quality.

Findings

Significantly reduces optimization oscillations.

Mitigates inter-phase knowledge gaps.

Achieves up to 13.5% improvement on Flickr30k.

Abstract

Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST) -- a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: https://github.com/Previsior/PTM-ST.