Real-Time Human Action Recognition on Embedded Platforms

TL;DR

This paper presents RT-HARE, a real-time human action recognition system optimized for embedded platforms, featuring a novel motion feature extractor that achieves 30 fps with high accuracy.

Contribution

The work introduces the Integrated Motion Feature Extractor (IMFE), a new neural network architecture that significantly reduces latency in HAR pipelines on embedded devices.

Findings

RT-HARE achieves 30 fps on Nvidia Jetson Xavier NX.

IMFE drastically improves latency over standard optical flow methods.

The system maintains high recognition accuracy in real-time conditions.

Abstract

With advancements in computer vision and deep learning, video-based human action recognition (HAR) has become practical. However, due to the complexity of the computation pipeline, running HAR on live video streams incurs excessive delays on embedded platforms. This work tackles the real-time performance challenges of HAR with four contributions: 1) an experimental study identifying a standard Optical Flow (OF) extraction technique as the latency bottleneck in a state-of-the-art HAR pipeline, 2) an exploration of the latency-accuracy tradeoff between the standard and deep learning approaches to OF extraction, which highlights the need for a novel, efficient motion feature extractor, 3) the design of Integrated Motion Feature Extractor (IMFE), a novel single-shot neural network architecture for motion feature extraction with drastic improvement in latency, 4) the development of RT-HARE, a real-time HAR system tailored for embedded platforms. Experimental results on an Nvidia Jetson Xavier NX platform demonstrated that RT-HARE realizes real-time HAR at a video frame rate of 30 frames per second while delivering high levels of recognition accuracy.