MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer

TL;DR

MVAFormer introduces a transformer-based multi-view cooperation module for spatio-temporal action recognition, effectively utilizing feature maps to preserve spatial information and improve recognition accuracy in multi-view, sequential settings.

Contribution

The paper proposes a novel transformer-based cooperation module that uses feature maps for multi-view spatio-temporal action recognition, addressing limitations of previous methods.

Findings

Outperforms baselines by approximately 4.4 F-measure points.

Effectively models relationships between multiple views.

Preserves spatial information through feature map utilization.

Abstract

Multi-view action recognition aims to recognize human actions using multiple camera views and deals with occlusion caused by obstacles or crowds. In this task, cooperation among views, which generates a joint representation by combining multiple views, is vital. Previous studies have explored promising cooperation methods for improving performance. However, since their methods focus only on the task setting of recognizing a single action from an entire video, they are not applicable to the recently popular spatio-temporal action recognition~(STAR) setting, in which each person's action is recognized sequentially. To address this problem, this paper proposes a multi-view action recognition method for the STAR setting, called MVAFormer. In MVAFormer, we introduce a novel transformer-based cooperation module among views. In contrast to previous studies, which utilize embedding vectors with lost spatial information, our module utilizes the feature map for effective cooperation in the STAR setting, which preserves the spatial information. Furthermore, in our module, we divide the self-attention for the same and different views to model the relationship between multiple views effectively. The results of experiments using a newly collected dataset demonstrate that MVAFormer outperforms the comparison baselines by approximately $4.4$ points on the F-measure.