CAST: Cross-Attention in Space and Time for Video Action Recognition

TL;DR

CAST introduces a novel cross-attention architecture for video action recognition that effectively balances spatial and temporal understanding using only RGB input, leading to improved performance across multiple benchmarks.

Contribution

The paper proposes a new two-stream architecture with a bottleneck cross-attention mechanism enabling better spatial-temporal information exchange in video recognition.

Findings

Consistently outperforms existing methods on EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400.

Achieves balanced spatio-temporal understanding with only RGB input.

Demonstrates robustness across diverse video datasets.

Abstract

Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics.