Boosting Point-supervised Temporal Action Localization via Text Refinement and Alignment

TL;DR

This paper introduces a novel framework that enhances point-supervised temporal action localization by integrating textual descriptions through refinement and alignment modules, significantly improving localization accuracy.

Contribution

The proposed TRA framework uniquely combines visual and textual features using new modules, enabling better semantic understanding and more precise action localization in videos.

Findings

Outperforms state-of-the-art methods on five benchmarks.

Effectively reduces modality gap via contrastive learning.

Operates efficiently on a single high-end GPU.

Abstract

Recently, point-supervised temporal action localization has gained significant attention for its effective balance between labeling costs and localization accuracy. However, current methods only consider features from visual inputs, neglecting helpful semantic information from the text side. To address this issue, we propose a Text Refinement and Alignment (TRA) framework that effectively utilizes textual features from visual descriptions to complement the visual features as they are semantically rich. This is achieved by designing two new modules for the original point-supervised framework: a Point-based Text Refinement module (PTR) and a Point-based Multimodal Alignment module (PMA). Specifically, we first generate descriptions for video frames using a pre-trained multimodal model. Next, PTR refines the initial descriptions by leveraging point annotations together with multiple pre-trained models. PMA then projects all features into a unified semantic space and leverages a point-level multimodal feature contrastive learning to reduce the gap between visual and linguistic modalities. Last, the enhanced multi-modal features are fed into the action detector for precise localization. Extensive experimental results on five widely used benchmarks demonstrate the favorable performance of our proposed framework compared to several state-of-the-art methods. Moreover, our computational overhead analysis shows that the framework can run on a single 24 GB RTX 3090 GPU, indicating its practicality and scalability.