See or Hear? Exploring the Effect of Visual and Audio Hints and Gaze-assisted Task Feedback for Visual Search Tasks in Augmented Reality

TL;DR

This study investigates how visual and audio hints, combined with gaze-assisted feedback, influence the efficiency of visual search tasks in augmented reality, demonstrating that hints and feedback can improve search performance.

Contribution

The paper introduces a novel AR approach integrating visual/audio hints and gaze-assisted feedback, and empirically evaluates their combined effect on search efficiency.

Findings

Gaze-assisted feedback significantly improves search performance.

Visual and audio hints together conditionally enhance search efficiency.

Hints alone have limited impact without feedback.

Abstract

Augmented reality (AR) is emerging in visual search tasks for increasingly immersive interactions with virtual objects. We propose an AR approach providing visual and audio hints along with gaze-assisted instant post-task feedback for search tasks based on mobile head-mounted display (HMD). The target case was a book-searching task, in which we aimed to explore the effect of the hints together with the task feedback with two hypotheses. H1: Since visual and audio hints can positively affect AR search tasks, the combination outperforms the individuals. H2: The gaze-assisted instant post-task feedback can positively affect AR search tasks. The proof-of-concept was demonstrated by an AR app in HMD and a comprehensive user study (n=96) consisting of two sub-studies, Study I (n=48) without task feedback and Study II (n=48) with task feedback. Following quantitative and qualitative analysis, our results partially verified H1 and completely verified H2, enabling us to conclude that the synthesis of visual and audio hints conditionally improves the AR visual search task efficiency when coupled with task feedback.