Scale-Gest: Scalable Model-Space Synthesis and Runtime Selection for On-Device Gesture Detection

TL;DR

Scale-Gest is an adaptive gesture detection framework that dynamically selects from a family of tiny-YOLO models to optimize performance, energy, and latency on mobile devices in real-time scenarios.

Contribution

It introduces a run-time adaptive system with device-calibrated ACE profiles and a motion-aware ROI gate for efficient on-device gesture detection under constraints.

Findings

Reduces energy per frame by 4x on a laptop

Maintains high gesture detection F1 score of 0.8-0.9

Achieves low latency of 6 ms per frame

Abstract

Realizing on-device ML-based gesture detection under tight real-time performance, energy and memory constraints is challenging, especially when considering mobile devices with varying battery-power levels. Existing EdgeAI deployments typically rely on a single fixed detector, limiting optimization opportunities. We present Scale-Gest, a novel run-time adaptive gesture detection framework that expands the detector space into a dense family of tiny-YOLO architectures. We introduce multiple novel device-calibrated ACE (Accuracy-Complexity-Energy) profiles by analyzing different model-resolution-stride operating points. A lightweight run-time controller selects an appropriate ACE mode under user-defined and battery constraints, while a motion-aware hand-gesture-tracking ROI gate crops the input for reduced complexity detection. To evaluate performance of our system in real-world car driving scenarios, we introduce a temporally-annotated Driver Simulated Gesture (DSG-18) dataset. Scale-Gest maintains event-level F1 while significantly reducing energy and latency compared to single-detector approaches. On a battery-powered laptop running gesture streams, our ACE controller reduces per-frame energy by 4x (from 6.9 mJ to 1.6 mJ) while maintaining high gesture-detection performance (event-level F1 = 0.8-0.9) and low mean latency (6 ms).