The Role of Video Generation in Enhancing Data-Limited Action Understanding

TL;DR

This paper introduces a text-to-video diffusion transformer to generate annotated data, significantly improving data-limited action understanding and achieving state-of-the-art zero-shot action recognition performance.

Contribution

The paper presents a novel data augmentation method using text-to-video diffusion transformers, along with strategies to enhance sample informativeness and mitigate low-quality data effects.

Findings

Achieved state-of-the-art zero-shot action recognition results.

Generated realistic annotated video data on an infinite scale.

Enhanced training with information and uncertainty-based strategies.

Abstract

Video action understanding tasks in real-world scenarios always suffer data limitations. In this paper, we address the data-limited action understanding problem by bridging data scarcity. We propose a novel method that employs a text-to-video diffusion transformer to generate annotated data for model training. This paradigm enables the generation of realistic annotated data on an infinite scale without human intervention. We proposed the information enhancement strategy and the uncertainty-based label smoothing tailored to generate sample training. Through quantitative and qualitative analysis, we observed that real samples generally contain a richer level of information than generated samples. Based on this observation, the information enhancement strategy is proposed to enhance the informative content of the generated samples from two aspects: the environments and the characters. Furthermore, we observed that some low-quality generated samples might negatively affect model training. To address this, we devised the uncertainty-based label smoothing strategy to increase the smoothing of these samples, thus reducing their impact. We demonstrate the effectiveness of the proposed method on four datasets across five tasks and achieve state-of-the-art performance for zero-shot action recognition.