Lattice Long Short-Term Memory for Human Action Recognition

TL;DR

This paper introduces Lattice-LSTM, an advanced neural network model that better captures long-term and non-stationary motion dynamics in video-based human action recognition, outperforming existing methods.

Contribution

The paper proposes Lattice-LSTM, which learns independent spatial hidden state transitions, and a multi-modal training approach that jointly trains input and forget gates across modalities.

Findings

Outperforms existing LSTM and CNN-based methods on UCF-101 and HMDB-51 datasets.

Effectively models long-term, non-stationary motion dynamics in videos.

Enhances human action recognition accuracy with comparable model complexity.

Abstract

Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.