Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics

TL;DR

This paper introduces DAViD, a reinforcement learning-based method for adaptive temporal resolution in video dataset distillation, significantly improving efficiency and performance by considering video semantics.

Contribution

The paper presents the first adaptive temporal resolution approach for video dataset distillation using reinforcement learning, addressing semantic variability in videos.

Findings

Outperforms existing dataset distillation methods on video datasets

Significantly improves model performance with adaptive temporal resolution

Demonstrates the effectiveness of semantic-aware video distillation

Abstract

With the rapid development of vision tasks and the scaling on datasets and models, redundancy reduction in vision datasets has become a key area of research. To address this issue, dataset distillation (DD) has emerged as a promising approach to generating highly compact synthetic datasets with significantly less redundancy while preserving essential information. However, while DD has been extensively studied for image datasets, DD on video datasets remains underexplored. Video datasets present unique challenges due to the presence of temporal information and varying levels of redundancy across different classes. Existing DD approaches assume a uniform level of temporal redundancy across all different video semantics, which limits their effectiveness on video datasets. In this work, we propose Dynamic-Aware Video Distillation (DAViD), a Reinforcement Learning (RL) approach to predict the optimal Temporal Resolution of the synthetic videos. A teacher-in-the-loop reward function is proposed to update the RL agent policy. To the best of our knowledge, this is the first study to introduce adaptive temporal resolution based on video semantics in video dataset distillation. Our approach significantly outperforms existing DD methods, demonstrating substantial improvements in performance. This work paves the way for future research on more efficient and semantic-adaptive video dataset distillation research.