Spatio-Temporal Context for Action Detection

TL;DR

This paper introduces a novel attention-based approach that models non-aggregated spatio-temporal interactions for action detection, leading to improved accuracy on the AVA dataset.

Contribution

It proposes two cross attention blocks to effectively model spatial relations and short-range temporal interactions in action detection.

Findings

Outperforms existing methods by +0.31 mAP on AVA dataset

Effectively models spatio-temporal relations between scene elements

Demonstrates the importance of non-aggregated temporal information

Abstract

Research in action detection has grown in the recentyears, as it plays a key role in video understanding. Modelling the interactions (either spatial or temporal) between actors and their context has proven to be essential for this task. While recent works use spatial features with aggregated temporal information, this work proposes to use non-aggregated temporal information. This is done by adding an attention based method that leverages spatio-temporal interactions between elements in the scene along the clip.The main contribution of this work is the introduction of two cross attention blocks to effectively model the spatial relations and capture short range temporal interactions.Experiments on the AVA dataset show the advantages of the proposed approach that models spatio-temporal relations between relevant elements in the scene, outperforming other methods that model actor interactions with their context by +0.31 mAP.