EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition

TL;DR

EPAM-Net is a novel, efficient multimodal network for video action recognition that significantly reduces computational costs while outperforming state-of-the-art methods on multiple datasets.

Contribution

The paper introduces EPAM-Net, combining pose-driven attention with an efficient X-ShiftNet architecture to improve action recognition with reduced FLOPs and parameters.

Findings

Outperforms state-of-the-art on multiple datasets

Achieves up to 72.8x reduction in FLOPs

Reduces network parameters by up to 48.6x

Abstract

Existing multimodal-based human action recognition approaches are computationally intensive, limiting their deployment in real-time applications. In this work, we present a novel and efficient pose-driven attention-guided multimodal network (EPAM-Net) for action recognition in videos. Specifically, we propose eXpand temporal Shift (X-ShiftNet) convolutional architectures for RGB and pose streams to capture spatio-temporal features from RGB videos and their skeleton sequences. The X-ShiftNet tackles the high computational cost of the 3D CNNs by integrating the Temporal Shift Module (TSM) into an efficient 2D CNN, enabling efficient spatiotemporal learning. Then skeleton features are utilized to guide the visual network stream, focusing on keyframes and their salient spatial regions using the proposed spatial-temporal attention block. Finally, the predictions of the two streams are fused for final classification. The experimental results show that our method, with a significant reduction in floating-point operations (FLOPs), outperforms and competes with the state-of-the-art methods on NTU RGB-D 60, NTU RGB-D 120, PKU-MMD, and Toyota SmartHome datasets. The proposed EPAM-Net provides up to a 72.8x reduction in FLOPs and up to a 48.6x reduction in the number of network parameters. The code will be available at https://github.com/ahmed-nady/Multimodal-Action-Recognition.