Hierarchical Multi-Stage Transformer Architecture for Context-Aware Temporal Action Localization

TL;DR

This paper introduces PCL-Former, a hierarchical multi-stage transformer architecture for temporal action localization, which effectively identifies, classifies, and precisely localizes actions in untrimmed videos, outperforming existing methods.

Contribution

The paper proposes a novel hierarchical transformer architecture with dedicated modules for proposal, classification, and localization, advancing the state-of-the-art in temporal action localization.

Findings

Outperforms state-of-the-art on THUMOS-14, ActivityNet-1.3, and HACS datasets.

Each module's impact validated through ablation studies.

Achieves 2.8%, 1.2%, and 4.8% improvements respectively.

Abstract

Inspired by the recent success of transformers and multi-stage architectures in video recognition and object detection domains. We thoroughly explore the rich spatio-temporal properties of transformers within a multi-stage architecture paradigm for the temporal action localization (TAL) task. This exploration led to the development of a hierarchical multi-stage transformer architecture called PCL-Former, where each subtask is handled by a dedicated transformer module with a specialized loss function. Specifically, the Proposal-Former identifies candidate segments in an untrimmed video that may contain actions, the Classification-Former classifies the action categories within those segments, and the Localization-Former precisely predicts the temporal boundaries (i.e., start and end) of the action instances. To evaluate the performance of our method, we have conducted extensive experiments on three challenging benchmark datasets: THUMOS-14, ActivityNet-1.3, and HACS Segments. We also conducted detailed ablation experiments to assess the impact of each individual module of our PCL-Former. The obtained quantitative results validate the effectiveness of the proposed PCL-Former, outperforming state-of-the-art TAL approaches by 2.8%, 1.2%, and 4.8% on THUMOS14, ActivityNet-1.3, and HACS datasets, respectively.