GestARLite: An On-Device Pointing Finger Based Gestural Interface for Smartphones and Video See-Through Head-Mounts
Varun Jain, Gaurav Garg, Ramakrishna Perla, Ramya Hebbalaguppe

TL;DR
This paper introduces a lightweight, real-time hand gesture recognition framework for wearable devices that combines deep learning models for localization and classification, enabling intuitive interaction in mixed reality applications.
Contribution
It presents a novel on-device gesture recognition system using a cascade of lightweight deep learning models optimized for smartphones and wearable MR devices.
Findings
Achieves 80% classification accuracy on EgoGestAR dataset
Operates in real-time with 0.12 seconds latency
Works effectively on frugal wearable devices like Google Cardboard
Abstract
Hand gestures form an intuitive means of interaction in Mixed Reality (MR) applications. However, accurate gesture recognition can be achieved only through state-of-the-art deep learning models or with the use of expensive sensors. Despite the robustness of these deep learning models, they are generally computationally expensive and obtaining real-time performance on-device is still a challenge. To this end, we propose a novel lightweight hand gesture recognition framework that works in First Person View for wearable devices. The models are trained on a GPU machine and ported on an Android smartphone for its use with frugal wearable devices such as the Google Cardboard and VR Box. The proposed hand gesture recognition framework is driven by a cascade of state-of-the-art deep learning models: MobileNetV2 for hand localisation, our custom fingertip regression architecture followed by a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHand Gesture Recognition Systems · Human Pose and Action Recognition · Tactile and Sensory Interactions
MethodsDepthwise Convolution · Pointwise Convolution · Depthwise Separable Convolution · Batch Normalization · Inverted Residual Block · Average Pooling · 1x1 Convolution · Convolution · Tether Customer Service Number +1-833-534-1729
