Boosting Weakly-Supervised Temporal Action Localization with Text Information

TL;DR

This paper introduces a novel approach leveraging text descriptions to improve weakly-supervised temporal action localization by enhancing discriminative and generative objectives, leading to state-of-the-art results.

Contribution

The paper proposes a new Text-Segment Mining mechanism and a Video-text Language Completion objective to better utilize text information in WTAL, improving localization accuracy.

Findings

Achieved state-of-the-art performance on THUMOS14 and ActivityNet1.3.

Method can be seamlessly integrated into existing WTAL models.

Significant performance improvements demonstrated across benchmarks.

Abstract

Due to the lack of temporal annotation, current Weakly-supervised Temporal Action Localization (WTAL) methods are generally stuck into over-complete or incomplete localization. In this paper, we aim to leverage the text information to boost WTAL from two aspects, i.e., (a) the discriminative objective to enlarge the inter-class difference, thus reducing the over-complete; (b) the generative objective to enhance the intra-class integrity, thus finding more complete temporal boundaries. For the discriminative objective, we propose a Text-Segment Mining (TSM) mechanism, which constructs a text description based on the action class label, and regards the text as the query to mine all class-related segments. Without the temporal annotation of actions, TSM compares the text query with the entire videos across the dataset to mine the best matching segments while ignoring irrelevant ones. Due to the shared sub-actions in different categories of videos, merely applying TSM is too strict to neglect the semantic-related segments, which results in incomplete localization. We further introduce a generative objective named Video-text Language Completion (VLC), which focuses on all semantic-related segments from videos to complete the text sentence. We achieve the state-of-the-art performance on THUMOS14 and ActivityNet1.3. Surprisingly, we also find our proposed method can be seamlessly applied to existing methods, and improve their performances with a clear margin. The code is available at https://github.com/lgzlIlIlI/Boosting-WTAL.