Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition

TL;DR

This paper introduces a novel body joint guided pooling method for 3D CNNs in action recognition, improving feature discriminability by leveraging body joint positions during feature pooling.

Contribution

It proposes a body joint guided feature pooling approach and a two-stream bilinear model that enhances 3D CNNs for action recognition without relying on precise skeleton detection.

Findings

Effective pooling guided by body joints improves recognition accuracy.

The bilinear model captures spatio-temporal features and guidance simultaneously.

Promising results on real-world datasets demonstrate the method's effectiveness.

Abstract

Three dimensional convolutional neural networks (3D CNNs) have been established as a powerful tool to simultaneously learn features from both spatial and temporal dimensions, which is suitable to be applied to video-based action recognition. In this work, we propose not to directly use the activations of fully-connected layers of a 3D CNN as the video feature, but to use selective convolutional layer activations to form a discriminative descriptor for video. It pools the feature on the convolutional layers under the guidance of body joint positions. Two schemes of mapping body joints into convolutional feature maps for pooling are discussed. The body joint positions can be obtained from any off-the-shelf skeleton estimation algorithm. The helpfulness of the body joint guided feature pooling with inaccurate skeleton estimation is systematically evaluated. To make it end-to-end and do not rely on any sophisticated body joint detection algorithm, we further propose a two-stream bilinear model which can learn the guidance from the body joints and capture the spatio-temporal features simultaneously. In this model, the body joint guided feature pooling is conveniently formulated as a bilinear product operation. Experimental results on three real-world datasets demonstrate the effectiveness of body joint guided pooling which achieves promising performance.