Two-Stream temporal transformer for video action classification

TL;DR

This paper introduces a novel two-stream transformer model for video action classification that effectively captures spatio-temporal features from content and optical flow, achieving high accuracy on standard datasets.

Contribution

The study presents a new two-stream transformer architecture that leverages self-attention for joint analysis of content and motion in videos, advancing video understanding methods.

Findings

Achieves excellent classification accuracy on three human activity datasets.

Effectively models relationships between spatial and temporal features.

Demonstrates the effectiveness of transformer-based models in video action recognition.

Abstract

Motion representation plays an important role in video understanding and has many applications including action recognition, robot and autonomous guidance or others. Lately, transformer networks, through their self-attention mechanism capabilities, have proved their efficiency in many applications. In this study, we introduce a new two-stream transformer video classifier, which extracts spatio-temporal information from content and optical flow representing movement information. The proposed model identifies self-attention features across the joint optical flow and temporal frame domain and represents their relationships within the transformer encoder mechanism. The experimental results show that our proposed methodology provides excellent classification results on three well-known video datasets of human activities.