Squeeze-and-Excitation on Spatial and Temporal Deep Feature Space for Action Recognition

TL;DR

This paper introduces SE-LRCN, a novel deep learning model that combines squeeze-and-excitation mechanisms with ResNet and LSTM to effectively capture spatial and temporal features for human action recognition, achieving competitive results on benchmark datasets.

Contribution

The paper proposes SE-LRCN, integrating squeeze-and-excitation modules into ResNet and LSTM for improved spatial-temporal feature modeling in action recognition.

Findings

Achieves competitive accuracy on HMDB51 and UCF101 datasets.

Enhances feature channel importance for spatial and temporal data.

Demonstrates effectiveness of squeeze-and-excitation in deep action recognition models.

Abstract

Spatial and temporal features are two key and complementary information for human action recognition. In order to make full use of the intra-frame spatial characteristics and inter-frame temporal relationships, we propose the Squeeze-and-Excitation Long-term Recurrent Convolutional Networks (SE-LRCN) for human action recognition. The Squeeze and Excitation operations are used to implement the feature recalibration. In SE-LRCN, Squeeze-and-Excitation ResNet-34 (SE-ResNet-34) network is adopted to extract spatial features to enhance the dependencies and importance of feature channels of pixel granularity. We also propose the Squeeze-and-Excitation Long Short-Term Memory (SE-LSTM) network to model the temporal relationship, and to enhance the dependencies and importance of feature channels of frame granularity. We evaluate the proposed model on two challenging benchmarks, HMDB51 and UCF101, and the proposed SE-LRCN achieves the competitive results with the state-of-the-art.