Gaze Gestures and Their Applications in human-computer interaction with a head-mounted display

TL;DR

This paper introduces two neural network models for gaze gesture recognition using head-mounted displays, demonstrating high accuracy and robustness for human-computer interaction in various environments.

Contribution

The paper presents novel UnityEyes-based CNN models UEGazeNet and UEGazeNet* for gaze gesture classification, along with a new dataset of eye-painting gestures for HCI applications.

Findings

UEGazeNet outperforms state-of-the-art networks by 52-67%

Achieves an average recognition rate of 96.71% on the GTgestures dataset

Effective indoor and outdoor gaze gesture recognition demonstrated

Abstract

A head-mounted display (HMD) is a portable and interactive display device. With the development of 5G technology, it may become a general-purpose computing platform in the future. Human-computer interaction (HCI) technology for HMDs has also been of significant interest in recent years. In addition to tracking gestures and speech, tracking human eyes as a means of interaction is highly effective. In this paper, we propose two UnityEyes-based convolutional neural network models, UEGazeNet and UEGazeNet*, which can be used for input images with low resolution and high resolution, respectively. These models can perform rapid interactions by classifying gaze trajectories (GTs), and a GTgestures dataset containing data for 10,200 "eye-painting gestures" collected from 15 individuals is established with our gaze-tracking method. We evaluated the performance both indoors and outdoors and the UEGazeNet can obtaine results 52\% and 67\% better than those of state-of-the-art networks. The generalizability of our GTgestures dataset using a variety of gaze-tracking models is evaluated, and an average recognition rate of 96.71\% is obtained by our method.