ProTAL: A Drag-and-Link Video Programming Framework for Temporal Action Localization

TL;DR

ProTAL introduces a user-friendly drag-and-link framework for defining key events in videos, enabling efficient label generation for temporal action localization and reducing manual annotation efforts.

Contribution

It presents a novel video programming framework that simplifies defining complex actions and generates training labels for TAL using semi-supervised learning.

Findings

Effective label generation for unlabelled videos

User study shows ease of defining key events

Improves TAL model training with less manual annotation

Abstract

Temporal Action Localization (TAL) aims to detect the start and end timestamps of actions in a video. However, the training of TAL models requires a substantial amount of manually annotated data. Data programming is an efficient method to create training labels with a series of human-defined labeling functions. However, its application in TAL faces difficulties of defining complex actions in the context of temporal video frames. In this paper, we propose ProTAL, a drag-and-link video programming framework for TAL. ProTAL enables users to define \textbf{key events} by dragging nodes representing body parts and objects and linking them to constrain the relations (direction, distance, etc.). These definitions are used to generate action labels for large-scale unlabelled videos. A semi-supervised method is then employed to train TAL models with such labels. We demonstrate the effectiveness of ProTAL through a usage scenario and a user study, providing insights into designing video programming framework.