Visual Attribute-augmented Three-dimensional Convolutional Neural Network for Enhanced Human Action Recognition

TL;DR

This paper introduces a novel visual attribute-augmented 3D CNN framework that integrates object detection and encoding to improve human action recognition accuracy in videos, achieving state-of-the-art results.

Contribution

The paper proposes a new multi-stream 3D CNN architecture incorporating visual attributes, enhancing action recognition performance over existing methods.

Findings

Achieves state-of-the-art accuracy on HMDB51 dataset

Outperforms previous methods on UCF101 dataset

Efficient integration of visual attributes improves recognition results

Abstract

Visual attributes in individual video frames, such as the presence of characteristic objects and scenes, offer substantial information for action recognition in videos. With individual 2D video frame as input, visual attributes extraction could be achieved effectively and efficiently with more sophisticated convolutional neural network than current 3D CNNs with spatio-temporal filters, thanks to fewer parameters in 2D CNNs. In this paper, the integration of visual attributes (including detection, encoding and classification) into multi-stream 3D CNN is proposed for action recognition in trimmed videos, with the proposed visual Attribute-augmented 3D CNN (A3D) framework. The visual attribute pipeline includes an object detection network, an attributes encoding network and a classification network. Our proposed A3D framework achieves state-of-the-art performance on both the HMDB51 and the UCF101 datasets.