On the Importance of Spatial Relations for Few-shot Action Recognition

TL;DR

This paper emphasizes the significance of spatial relations in few-shot action recognition and introduces a novel Spatial Alignment Cross Transformer (SA-CT) that effectively leverages spatial and temporal information, achieving competitive results.

Contribution

The paper proposes a new SA-CT model that focuses on spatial relations and integrates temporal data, advancing few-shot action recognition methods.

Findings

SA-CT performs comparably to temporal-based methods without using temporal info.

Adding the Temporal Mixer improves video representation and overall accuracy.

Large-scale pretrained models enhance few-shot action recognition performance.

Abstract

Deep learning has achieved great success in video recognition, yet still struggles to recognize novel actions when faced with only a few examples. To tackle this challenge, few-shot action recognition methods have been proposed to transfer knowledge from a source dataset to a novel target dataset with only one or a few labeled videos. However, existing methods mainly focus on modeling the temporal relations between the query and support videos while ignoring the spatial relations. In this paper, we find that the spatial misalignment between objects also occurs in videos, notably more common than the temporal inconsistency. We are thus motivated to investigate the importance of spatial relations and propose a more accurate few-shot action recognition method that leverages both spatial and temporal information. Particularly, a novel Spatial Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial relations and incorporates the temporal information is contributed. Experiments reveal that, even without using any temporal information, the performance of SA-CT is comparable to temporal based methods on 3/4 benchmarks. To further incorporate the temporal information, we propose a simple yet effective Temporal Mixer module. The Temporal Mixer enhances the video representation and improves the performance of the full SA-CT model, achieving very competitive results. In this work, we also exploit large-scale pretrained models for few-shot action recognition, providing useful insights for this research direction.