A Large-Scale Study on Video Action Dataset Condensation

TL;DR

This paper conducts a comprehensive large-scale study on video dataset condensation, exploring key aspects like temporal processing and evaluation protocols, and achieves state-of-the-art results on multiple action recognition datasets.

Contribution

It provides the first systematic analysis of video dataset condensation, proposing a unified evaluation protocol and adapting algorithms to the space-time domain.

Findings

Labeling methods significantly affect performance.

Sliding-window sampling effectively handles temporal data.

Distillation methods excel in challenging scenarios, while sample selection methods perform better in easier cases.

Abstract

Recently, dataset condensation has made significant progress in the image domain. Unlike images, videos possess an additional temporal dimension, which harbors considerable redundant information, making condensation even more crucial. However, video dataset condensation still remains an underexplored area. We aim to bridge this gap by providing a large-scale study with systematic design and fair comparison. Specifically, our work delves into three key aspects to provide valuable empirical insights: (1) temporal processing of video data, (2) the evaluation protocol for video dataset condensation, and (3) adaptation of condensation algorithms to the space-time domain. From this study, we derive several intriguing observations: (i) labeling methods greatly influence condensation performance, (ii) simple sliding-window sampling is effective for temporal processing, and (iii) dataset distillation methods perform better in challenging scenarios, while sample selection methods excel in easier ones. Furthermore, we propose a unified evaluation protocol for the fair comparison of different condensation algorithms and achieve state-of-the-art results on four widely-used action recognition datasets: HMDB51, UCF101, SSv2 and K400. Our code is available at https://github.com/MCG-NJU/Video-DC.