Efficient Human Vision Inspired Action Recognition using Adaptive Spatiotemporal Sampling

TL;DR

This paper introduces a human vision-inspired adaptive spatiotemporal sampling method for efficient action recognition on wearable devices, significantly improving speed with minimal accuracy loss.

Contribution

It proposes a novel context-aware sampling scheme inspired by human visual perception, enhancing efficiency over fixed sampling strategies.

Findings

Speeds up inference significantly

Maintains accuracy with minimal loss

Validated on EPIC-KITCHENS and UCF-101 datasets

Abstract

Adaptive sampling that exploits the spatiotemporal redundancy in videos is critical for always-on action recognition on wearable devices with limited computing and battery resources. The commonly used fixed sampling strategy is not context-aware and may under-sample the visual content, and thus adversely impacts both computation efficiency and accuracy. Inspired by the concepts of foveal vision and pre-attentive processing from the human visual perception mechanism, we introduce a novel adaptive spatiotemporal sampling scheme for efficient action recognition. Our system pre-scans the global scene context at low-resolution and decides to skip or request high-resolution features at salient regions for further processing. We validate the system on EPIC-KITCHENS and UCF-101 datasets for action recognition, and show that our proposed approach can greatly speed up inference with a tolerable loss of accuracy compared with those from state-of-the-art baselines. Source code is available in https://github.com/knmac/adaptive_spatiotemporal.