Heatmap Pooling Network for Action Recognition from RGB Videos

TL;DR

The paper introduces HP-Net, a novel heatmap pooling network that improves human action recognition in RGB videos by extracting robust, information-rich features and integrating multimodal data, outperforming existing methods.

Contribution

The paper proposes a heatmap pooling network with feedback pooling, spatial-motion co-learning, and text refinement modules for enhanced action recognition from RGB videos.

Findings

Outperforms existing action recognition methods on multiple benchmarks.

Demonstrates robustness and efficiency in feature extraction.

Effectively integrates multimodal data for improved accuracy.

Abstract

Human action recognition (HAR) in videos has garnered widespread attention due to the rich information in RGB videos. Nevertheless, existing methods for extracting deep features from RGB videos face challenges such as information redundancy, susceptibility to noise and high storage costs. To address these issues and fully harness the useful information in videos, we propose a novel heatmap pooling network (HP-Net) for action recognition from videos, which extracts information-rich, robust and concise pooled features of the human body in videos through a feedback pooling module. The extracted pooled features demonstrate obvious performance advantages over the previously obtained pose data and heatmap features from videos. In addition, we design a spatial-motion co-learning module and a text refinement modulation module to integrate the extracted pooled features with other multimodal data, enabling more robust action recognition. Extensive experiments on several benchmarks namely NTU RGB+D 60, NTU RGB+D 120, Toyota-Smarthome and UAV-Human consistently verify the effectiveness of our HP-Net, which outperforms the existing human action recognition methods. Our code is publicly available at: https://github.com/liujf69/HPNet-Action.