Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks

TL;DR

This paper introduces a factorized spatio-temporal convolutional network (FstCN) for human action recognition in videos, effectively capturing 3D signals by separating spatial and temporal learning, leading to improved performance on benchmark datasets.

Contribution

The paper proposes a novel factorized 3D CNN architecture with a transformation operator, enhancing training efficiency and accuracy in human action recognition tasks.

Findings

FstCN outperforms existing CNN-based methods on UCF-101 and HMDB-51 datasets.

FstCN achieves comparable results to methods using auxiliary training videos.

The approach effectively handles sequence alignment through a new sampling strategy.

Abstract

Human actions in video sequences are three-dimensional (3D) spatio-temporal signals characterizing both the visual appearance and motion dynamics of the involved humans and objects. Inspired by the success of convolutional neural networks (CNN) for image classification, recent attempts have been made to learn 3D CNNs for recognizing human actions in videos. However, partly due to the high complexity of training 3D convolution kernels and the need for large quantities of training videos, only limited success has been reported. This has triggered us to investigate in this paper a new deep architecture which can handle 3D signals more effectively. Specifically, we propose factorized spatio-temporal convolutional networks (FstCN) that factorize the original 3D convolution kernel learning as a sequential process of learning 2D spatial kernels in the lower layers (called spatial convolutional layers), followed by learning 1D temporal kernels in the upper layers (called temporal convolutional layers). We introduce a novel transformation and permutation operator to make factorization in FstCN possible. Moreover, to address the issue of sequence alignment, we propose an effective training and inference strategy based on sampling multiple video clips from a given action video sequence. We have tested FstCN on two commonly used benchmark datasets (UCF-101 and HMDB-51). Without using auxiliary training videos to boost the performance, FstCN outperforms existing CNN based methods and achieves comparable performance with a recent method that benefits from using auxiliary training videos.