Learn to cycle: Time-consistent feature discovery for action recognition

TL;DR

This paper introduces SRTG, a novel CNN block with an LSTM and temporal gate for more flexible and consistent feature discovery in action recognition, improving performance across multiple datasets.

Contribution

The paper proposes SRTG, a new approach that enhances temporal feature discovery by evaluating feature consistency, leading to improved action recognition accuracy.

Findings

Outperforms state-of-the-art on HACS, Moments in Time, UCF-101, HMDB-51

Achieves comparable results to top models on Kinetics-700

Minimal increase in computational cost

Abstract

Generalizing over temporal variations is a prerequisite for effective action recognition in videos. Despite significant advances in deep neural networks, it remains a challenge to focus on short-term discriminative motions in relation to the overall performance of an action. We address this challenge by allowing some flexibility in discovering relevant spatio-temporal features. We introduce Squeeze and Recursion Temporal Gates (SRTG), an approach that favors inputs with similar activations with potential temporal variations. We implement this idea with a novel CNN block that uses an LSTM to encapsulate feature dynamics, in conjunction with a temporal gate that is responsible for evaluating the consistency of the discovered dynamics and the modeled features. We show consistent improvement when using SRTG blocks, with only a minimal increase in the number of GFLOPs. On Kinetics-700, we perform on par with current state-of-the-art models, and outperform these on HACS, Moments in Time, UCF-101 and HMDB-51.