Video Dataset Condensation with Diffusion Models

TL;DR

This paper introduces a novel video dataset distillation method combining diffusion models, a specialized selection network, and a training-free clustering technique, achieving state-of-the-art performance with reduced computational costs.

Contribution

It presents a new approach for video dataset distillation using diffusion models, a tailored selection network, and a training-free clustering algorithm, significantly improving efficiency and effectiveness.

Findings

Achieves up to 10.61% performance improvement over state-of-the-art methods.

Demonstrates effectiveness across four benchmark datasets.

Reduces computational costs by generating videos once and using training-free selection.

Abstract

In recent years, the rapid expansion of dataset sizes and the increasing complexity of deep learning models have significantly escalated the demand for computational resources, both for data storage and model training. Dataset distillation has emerged as a promising solution to address this challenge by generating a compact synthetic dataset that retains the essential information from a large real dataset. However, existing methods often suffer from limited performance, particularly in the video domain. In this paper, we focus on video dataset distillation. We begin by employing a video diffusion model to generate synthetic videos. Since the videos are generated only once, this significantly reduces computational costs. Next, we introduce the Video Spatio-Temporal U-Net (VST-UNet), a model designed to select a diverse and informative subset of videos that effectively captures the characteristics of the original dataset. To further optimize computational efficiency, we explore a training-free clustering algorithm, Temporal-Aware Cluster-based Distillation (TAC-DT), to select representative videos without requiring additional training overhead. We validate the effectiveness of our approach through extensive experiments on four benchmark datasets, demonstrating performance improvements of up to \(10.61\%\) over the state-of-the-art. Our method consistently outperforms existing approaches across all datasets, establishing a new benchmark for video dataset distillation.