Real-time Egocentric Gesture Recognition on Mobile Head Mounted Displays

TL;DR

This paper presents a real-time egocentric gesture recognition system for mobile VR headsets, featuring a new data collection tool, a large annotated dataset, and a neural network that achieves high accuracy on mobile devices.

Contribution

Introduces a novel mixed-reality data collection tool, the largest egocentric gesture dataset to date, and a neural network optimized for real-time mobile performance.

Findings

Achieved over 76% precision in gesture recognition across 8 classes.

Created a dataset with more than 400,000 annotated frames.

Developed a neural network capable of real-time inference on mobile CPUs.

Abstract

Mobile virtual reality (VR) head mounted displays (HMD) have become popular among consumers in recent years. In this work, we demonstrate real-time egocentric hand gesture detection and localization on mobile HMDs. Our main contributions are: 1) A novel mixed-reality data collection tool to automatic annotate bounding boxes and gesture labels; 2) The largest-to-date egocentric hand gesture and bounding box dataset with more than 400,000 annotated frames; 3) A neural network that runs real time on modern mobile CPUs, and achieves higher than 76% precision on gesture recognition across 8 classes.