Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display

TL;DR

This paper introduces a unified, hardware-efficient neural network approach for recognizing in-place body actions and head gestures using only a head-mounted display, enabling more natural VR interactions without extra equipment.

Contribution

The paper proposes a two-stream 1-D CNN that recognizes multiple body actions and head gestures solely with an HMD, expanding interaction possibilities in VR without additional hardware.

Findings

Recognizes 10 body actions and 8 head gestures reliably.

Outperforms previous methods in hardware simplicity and recognition scope.

Demonstrates utility in VR locomotion and fitness applications.

Abstract

Body actions and head gestures are natural interfaces for interaction in virtual environments. Existing methods for in-place body action recognition often require hardware more than a head-mounted display (HMD), making body action interfaces difficult to be introduced to ordinary virtual reality (VR) users as they usually only possess an HMD. In addition, there lacks a unified solution to recognize in-place body actions and head gestures. This potentially hinders the exploration of the use of in-place body actions and head gestures for novel interaction experiences in virtual environments. We present a unified two-stream 1-D convolutional neural network (CNN) for recognition of body actions when a user performs walking-in-place (WIP) and for recognition of head gestures when a user stands still wearing only an HMD. Compared to previous approaches, our method does not require specialized hardware and/or additional tracking devices other than an HMD and can recognize a significantly larger number of body actions and head gestures than other existing methods. In total, ten in-place body actions and eight head gestures can be recognized with the proposed method, which makes this method a readily available body action interface (head gestures included) for interaction with virtual environments. We demonstrate one utility of the interface through a virtual locomotion task. Results show that the present body action interface is reliable in detecting body actions for the VR locomotion task but is physically demanding compared to a touch controller interface. The present body action interface is promising for new VR experiences and applications, especially for VR fitness applications where workouts are intended.