Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy

TL;DR

This paper introduces a multi-view attention consistency approach using Directed Gromov-Wasserstein Discrepancy to improve action recognition accuracy and reliability, incorporating Neural Radiance Fields for multi-view feature rendering, achieving state-of-the-art results.

Contribution

It proposes a novel multi-view attention consistency method with a new metric, and integrates Neural Radiance Fields into a Video Transformer-based action recognition model.

Findings

Achieves state-of-the-art performance on Jester, Something-Something V2, and Kinetics-400 datasets.

Demonstrates improved focus on relevant action subjects through attention consistency.

Validates the effectiveness of multi-view features and Neural Radiance Fields in action recognition.

Abstract

Action recognition has become one of the popular research topics in computer vision. There are various methods based on Convolutional Networks and self-attention mechanisms as Transformers to solve both spatial and temporal dimensions problems of action recognition tasks that achieve competitive performances. However, these methods lack a guarantee of the correctness of the action subject that the models give attention to, i.e., how to ensure an action recognition model focuses on the proper action subject to make a reasonable action prediction. In this paper, we propose a multi-view attention consistency method that computes the similarity between two attentions from two different views of the action videos using Directed Gromov-Wasserstein Discrepancy. Furthermore, our approach applies the idea of Neural Radiance Field to implicitly render the features from novel views when training on single-view datasets. Therefore, the contributions in this work are three-fold. Firstly, we introduce the multi-view attention consistency to solve the problem of reasonable prediction in action recognition. Secondly, we define a new metric for multi-view consistent attention using Directed Gromov-Wasserstein Discrepancy. Thirdly, we built an action recognition model based on Video Transformers and Neural Radiance Fields. Compared to the recent action recognition methods, the proposed approach achieves state-of-the-art results on three large-scale datasets, i.e., Jester, Something-Something V2, and Kinetics-400.