A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognition

TL;DR

This paper introduces EMIM, a module that enhances transformer-based action recognition by explicitly modeling motion through affinity matrices, significantly improving performance on motion-sensitive datasets.

Contribution

The paper proposes EMIM, a novel module that integrates explicit motion information mining into transformers using affinity matrices, addressing limitations on motion-sensitive datasets.

Findings

Outperforms state-of-the-art methods on multiple datasets

Significantly improves results on motion-sensitive datasets

Validates the effectiveness of explicit motion modeling

Abstract

Recently, action recognition has been dominated by transformer-based methods, thanks to their spatiotemporal contextual aggregation capacities. However, despite the significant progress achieved on scene-related datasets, they do not perform well on motion-sensitive datasets due to the lack of elaborate motion modeling designs. Meanwhile, we observe that the widely-used cost volume in traditional action recognition is highly similar to the affinity matrix defined in self-attention, but equipped with powerful motion modeling capacities. In light of this, we propose to integrate those effective motion modeling properties into the existing transformer in a unified and neat way, with the proposal of the Explicit Motion Information Mining module (EMIM). In EMIM, we propose to construct the desirable affinity matrix in a cost volume style, where the set of key candidate tokens is sampled from the query-based neighboring area in the next frame in a sliding-window manner. Then, the constructed affinity matrix is used to aggregate contextual information for appearance modeling and is converted into motion features for motion modeling as well. We validate the motion modeling capacities of our method on four widely-used datasets, and our method performs better than existing state-of-the-art approaches, especially on motion-sensitive datasets, i.e., Something-Something V1 & V2. Our project is available at https://github.com/PeiqinZhuang/EMIM .