EdgeOAR: Real-time Online Action Recognition On Edge Devices

TL;DR

EdgeOAR is a novel framework for real-time online action recognition on edge devices, significantly reducing latency and energy consumption while maintaining accuracy by using lightweight modules and innovative fusion techniques.

Contribution

The paper introduces EdgeOAR, a new framework with a task-specific feature enhancement module and fusion strategies tailored for online action recognition on resource-limited edge devices.

Findings

Reduces latency by 99.23% on UCF-101

Lowers energy consumption by 99.28%

Maintains adequate accuracy on edge devices

Abstract

This paper addresses the challenges of Online Action Recognition (OAR), a framework that involves instantaneous analysis and classification of behaviors in video streams. OAR must operate under stringent latency constraints, making it an indispensable component for real-time feedback for edge computing. Existing methods, which typically rely on the processing of entire video clips, fall short in scenarios requiring immediate recognition. To address this, we designed EdgeOAR, a novel framework specifically designed for OAR on edge devices. EdgeOAR includes the Early Exit-oriented Task-specific Feature Enhancement Module (TFEM), which comprises lightweight submodules to optimize features in both temporal and spatial dimensions. We design an iterative training method to enable TFEM learning features from the beginning of the video. Additionally, EdgeOAR includes an Inverse Information Entropy (IIE) and Modality Consistency (MC)-driven fusion module to fuse features and make better exit decisions. This design overcomes the two main challenges: robust modeling of spatio-temporal action representations with limited initial frames in online video streams and balancing accuracy and efficiency on resource-constrained edge devices. Experiments show that on the UCF-101 dataset, our method EdgeOAR reduces latency by 99.23% and energy consumption by 99.28% compared to state-of-the-art (SOTA) method. And achieves an adequate accuracy on edge devices.