Temporal Transformer Networks with Self-Supervision for Action Recognition

TL;DR

This paper introduces a Temporal Transformer Network with Self-supervision (TTSN) that models long-range temporal dependencies and reverses motion sequences, significantly improving action recognition performance on multiple datasets.

Contribution

The paper presents a novel TTSN architecture combining a temporal transformer and a self-supervision module with sequence reversal, enhancing motion feature modeling and generalization.

Findings

Achieves state-of-the-art results on HMDB51, UCF101, and Something-something V1 datasets.

Effectively models non-linear temporal dependencies among frames.

Improves robustness by reversing frame sequences for self-supervision.

Abstract

In recent years, 2D Convolutional Networks-based video action recognition has encouragingly gained wide popularity; However, constrained by the lack of long-range non-linear temporal relation modeling and reverse motion information modeling, the performance of existing models is, therefore, undercut seriously. To address this urgent problem, we introduce a startling Temporal Transformer Network with Self-supervision (TTSN). Our high-performance TTSN mainly consists of a temporal transformer module and a temporal sequence self-supervision module. Concisely speaking, we utilize the efficient temporal transformer module to model the non-linear temporal dependencies among non-local frames, which significantly enhances complex motion feature representations. The temporal sequence self-supervision module we employ unprecedentedly adopts the streamlined strategy of "random batch random channel" to reverse the sequence of video frames, allowing robust extractions of motion information representation from inversed temporal dimensions and improving the generalization capability of the model. Extensive experiments on three widely used datasets (HMDB51, UCF101, and Something-something V1) have conclusively demonstrated that our proposed TTSN is promising as it successfully achieves state-of-the-art performance for action recognition.