Probabilistic Temporal Masked Attention for Cross-view Online Action Detection

TL;DR

This paper introduces a Probabilistic Temporal Masked Attention model that improves cross-view online action detection by leveraging probabilistic representations and view-invariant features, achieving state-of-the-art results.

Contribution

The paper presents a novel PTMA model with probabilistic latent representations and a GRU-based attention mechanism for enhanced cross-view online action detection.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively models view-invariant features.

Improves generalization across unseen viewpoints.

Abstract

As a critical task in video sequence classification within computer vision, Online Action Detection (OAD) has garnered significant attention. The sensitivity of mainstream OAD models to varying video viewpoints often hampers their generalization when confronted with unseen sources. To address this limitation, we propose a novel Probabilistic Temporal Masked Attention (PTMA) model, which leverages probabilistic modeling to derive latent compressed representations of video frames in a cross-view setting. The PTMA model incorporates a GRU-based temporal masked attention (TMA) cell, which leverages these representations to effectively query the input video sequence, thereby enhancing information interaction and facilitating autoregressive frame-level video analysis. Additionally, multi-view information can be integrated into the probabilistic modeling to facilitate the extraction of view-invariant features. Experiments conducted under three evaluation protocols: cross-subject (cs), cross-view (cv), and cross-subject-view (csv) show that PTMA achieves state-of-the-art performance on the DAHLIA, IKEA ASM, and Breakfast datasets.