Vision Transformer with Cross-attention by Temporal Shift for Efficient Action Recognition

TL;DR

This paper introduces MSCA, a novel multi-head self/cross-attention mechanism for Vision Transformers, which enhances action recognition by better exploiting temporal information with minimal additional computation.

Contribution

It proposes MSCA, a new attention structure that replaces MSA in ViT, fully utilizing temporal shifts for improved action recognition performance.

Findings

MSCA-KV outperforms TokenShift by 0.1% on Kinetics400.

MSCA improves over standard ViT by 1.2%.

The method maintains the same computational cost as existing models.

Abstract

Feature shifts have been shown to be useful for action recognition with CNN-based models since Temporal Shift Module (TSM) was proposed. It is based on frame-wise feature extraction with late fusion, and layer features are shifted along the time direction for the temporal interaction. TokenShift, a recent model based on Vision Transformer (ViT), also uses the temporal feature shift mechanism, which, however, does not fully exploit the structure of Multi-head Self-Attention (MSA) in ViT. In this paper, we propose Multi-head Self/Cross-Attention (MSCA), which fully utilizes the attention structure. TokenShift is based on a frame-wise ViT with features temporally shifted with successive frames (at time t+1 and t-1). In contrast, the proposed MSCA replaces MSA in the frame-wise ViT, and some MSA heads attend to successive frames instead of the current frame. The computation cost is the same as the frame-wise ViT and TokenShift as it simply changes the target to which the attention is taken. There is a choice about which of key, query, and value are taken from the successive frames, then we experimentally compared these variants with Kinetics400. We also investigate other variants in which the proposed MSCA is used along the patch dimension of ViT, instead of the head dimension. Experimental results show that a variant, MSCA-KV, shows the best performance and is better than TokenShift by 0.1% and then ViT by 1.2%.