Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

TL;DR

This paper introduces a multi-task learning framework that leverages point supervision and self-supervised tasks to enhance temporal understanding for more accurate point-level weakly-supervised temporal action localization.

Contribution

It is the first to explicitly explore temporal consistency using self-supervised tasks in point-supervised action localization, improving model understanding of temporal relationships.

Findings

Outperforms state-of-the-art methods on four benchmarks.

Self-supervised tasks improve temporal understanding and localization accuracy.

Demonstrates the importance of modeling temporal relationships in weak supervision.

Abstract

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (\textit{i.e.}, labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.