Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization

TL;DR

This paper introduces a novel weakly-supervised framework for temporal action localization that uses dense pseudo-labels and sequence searching to improve the completeness of action predictions, achieving state-of-the-art results.

Contribution

The paper proposes a new method that generates pseudo background points and searches for complete action sequences, enhancing weakly-supervised action localization performance.

Findings

Significant performance improvements at high IoU thresholds.

Outperforms existing weakly-supervised methods on four benchmarks.

Achieves comparable results to fully-supervised methods with less annotation cost.

Abstract

We tackle the problem of localizing temporal intervals of actions with only a single frame label for each action instance for training. Owing to label sparsity, existing work fails to learn action completeness, resulting in fragmentary action predictions. In this paper, we propose a novel framework, where dense pseudo-labels are generated to provide completeness guidance for the model. Concretely, we first select pseudo background points to supplement point-level action labels. Then, by taking the points as seeds, we search for the optimal sequence that is likely to contain complete action instances while agreeing with the seeds. To learn completeness from the obtained sequence, we introduce two novel losses that contrast action instances with background ones in terms of action score and feature similarity, respectively. Experimental results demonstrate that our completeness guidance indeed helps the model to locate complete action instances, leading to large performance gains especially under high IoU thresholds. Moreover, we demonstrate the superiority of our method over existing state-of-the-art methods on four benchmarks: THUMOS'14, GTEA, BEOID, and ActivityNet. Notably, our method even performs comparably to recent fully-supervised methods, at the 6 times cheaper annotation cost. Our code is available at https://github.com/Pilhyeon.