Memory-Augmented Temporal Dynamic Learning for Action Recognition

TL;DR

This paper introduces a memory-augmented neural network for action recognition in videos, effectively modeling long-term motion dynamics by selectively updating an external memory, leading to improved accuracy on benchmark datasets.

Contribution

It proposes a novel memory-augmented learning framework with a differential memory controller for better long-term motion modeling in action recognition.

Findings

Achieved consistent improvements on UCF101 and HMDB51 datasets.

Demonstrated effective handling of long-duration motion dynamics.

Outperformed prior methods and baselines in experiments.

Abstract

Human actions captured in video sequences contain two crucial factors for action recognition, i.e., visual appearance and motion dynamics. To model these two aspects, Convolutional and Recurrent Neural Networks (CNNs and RNNs) are adopted in most existing successful methods for recognizing actions. However, CNN based methods are limited in modeling long-term motion dynamics. RNNs are able to learn temporal motion dynamics but lack effective ways to tackle unsteady dynamics in long-duration motion. In this work, we propose a memory-augmented temporal dynamic learning network, which learns to write the most evident information into an external memory module and ignore irrelevant ones. In particular, we present a differential memory controller to make a discrete decision on whether the external memory module should be updated with current feature. The discrete memory controller takes in the memory history, context embedding and current feature as inputs and controls information flow into the external memory module. Additionally, we train this discrete memory controller using straight-through estimator. We evaluate this end-to-end system on benchmark datasets (UCF101 and HMDB51) of human action recognition. The experimental results show consistent improvements on both datasets over prior works and our baselines.