PAT: Position-Aware Transformer for Dense Multi-Label Action Detection

TL;DR

PAT introduces a position-aware transformer that effectively models complex temporal action dependencies in videos by embedding relative positional encoding and utilizing a non-hierarchical multi-scale approach, leading to state-of-the-art results.

Contribution

The paper proposes a novel non-hierarchical transformer architecture with relative positional encoding for dense multi-label action detection in videos, addressing positional information loss in existing methods.

Findings

Achieves new state-of-the-art mAP on Charades and MultiTHUMOS datasets.

Improves previous results by 1.1% and 0.6% mAP respectively.

Extensive ablation studies validate the effectiveness of each component.

Abstract

We present PAT, a transformer-based network that learns complex temporal co-occurrence action dependencies in a video by exploiting multi-scale temporal features. In existing methods, the self-attention mechanism in transformers loses the temporal positional information, which is essential for robust action detection. To address this issue, we (i) embed relative positional encoding in the self-attention mechanism and (ii) exploit multi-scale temporal relationships by designing a novel non hierarchical network, in contrast to the recent transformer-based approaches that use a hierarchical structure. We argue that joining the self-attention mechanism with multiple sub-sampling processes in the hierarchical approaches results in increased loss of positional information. We evaluate the performance of our proposed approach on two challenging dense multi-label benchmark datasets, and show that PAT improves the current state-of-the-art result by 1.1% and 0.6% mAP on the Charades and MultiTHUMOS datasets, respectively, thereby achieving the new state-of-the-art mAP at 26.5% and 44.6%, respectively. We also perform extensive ablation studies to examine the impact of the different components of our proposed network.