An Efficient Spatio-Temporal Pyramid Transformer for Action Detection

TL;DR

This paper introduces an efficient hierarchical Spatio-Temporal Pyramid Transformer (STPT) that balances accuracy and computational efficiency for action detection in videos by combining local and global attention mechanisms.

Contribution

The paper proposes a novel hierarchical Transformer architecture that uses local window attention in early layers and global attention later, improving efficiency without sacrificing accuracy.

Findings

Achieves 53.6% mAP on THUMOS14 with only RGB input.

Outperforms I3D+AFSD RGB model by over 10%.

Uses 31% fewer GFLOPs than state-of-the-art AFSD.

Abstract

The task of action detection aims at deducing both the action category and localization of the start and end moment for each action instance in a long, untrimmed video. While vision Transformers have driven the recent advances in video understanding, it is non-trivial to design an efficient architecture for action detection due to the prohibitively expensive self-attentions over a long sequence of video clips. To this end, we present an efficient hierarchical Spatio-Temporal Pyramid Transformer (STPT) for action detection, building upon the fact that the early self-attention layers in Transformers still focus on local patterns. Specifically, we propose to use local window attention to encode rich local spatio-temporal representations in the early stages while applying global attention modules to capture long-term space-time dependencies in the later stages. In this way, our STPT can encode both locality and dependency with largely reduced redundancy, delivering a promising trade-off between accuracy and efficiency. For example, with only RGB input, the proposed STPT achieves 53.6% mAP on THUMOS14, surpassing I3D+AFSD RGB model by over 10% and performing favorably against state-of-the-art AFSD that uses additional flow features with 31% fewer GFLOPs, which serves as an effective and efficient end-to-end Transformer-based framework for action detection.