Predicting Human Performance in Vertical Hierarchical Menu Selection in Immersive AR Using Hand-gesture and Head-gaze

TL;DR

This paper presents a novel predictive model that estimates user performance in immersive AR hierarchical menu selection using hand-gesture and head-gaze, reducing the need for extensive user studies.

Contribution

The work introduces the first predictive model for user performance in immersive AR UI, incorporating semantic encoding and cognitive measures for accuracy.

Findings

High accuracy in predicting consumed endurance (CE) across diverse users.

Dynamic menu generation captures variability in hierarchical menu design.

Model reduces subjective bias by integrating cognitive performance data.

Abstract

There are currently limited guidelines on designing user interfaces (UI) for immersive augmented reality (AR) applications. Designers must reflect on their experience designing UI for desktop and mobile applications and conjecture how a UI will influence AR users' performance. In this work, we introduce a predictive model for determining users' performance for a target UI without the subsequent involvement of participants in user studies. The model is trained on participants' responses to objective performance measures such as consumed endurance (CE) and pointing time (PT) using hierarchical drop-down menus. Large variability in the depth and context of the menus is ensured by randomly and dynamically creating the hierarchical drop-down menus and associated user tasks from words contained in the lexical database WordNet. Subjective performance bias is reduced by incorporating the users' non-verbal standard performance WAIS-IV during the model training. The semantic information of the menu is encoded using the Universal Sentence Encoder. We present the results of a user study that demonstrates that the proposed predictive model achieves high accuracy in predicting the CE on hierarchical menus of users with various cognitive abilities. To the best of our knowledge, this is the first work on predicting CE in designing UI for immersive AR applications.