Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality

TL;DR

This paper introduces an explainable AI interface for gaze-based interactions in mixed reality, demonstrating that XAI improves user accuracy and adaptability in visual search tasks.

Contribution

It presents a real-time, multi-level XAI system for gaze-based XR interactions and evaluates its effectiveness in enhancing user performance and understanding.

Findings

XAI increased selection accuracy by 10.8%.

Participants adapted gaze behavior over time with XAI.

XAI facilitates better collaboration with model-driven XR interactions.

Abstract

Gaze-based interactions offer a potential way for users to naturally engage with mixed reality (XR) interfaces. Black-box machine learning models enabled higher accuracy for gaze-based interactions. However, due to the black-box nature of the model, users might not be able to understand and effectively adapt their gaze behaviour to achieve high quality interaction. We posit that explainable AI (XAI) techniques can facilitate understanding of and interaction with gaze-based model-driven system in XR. To study this, we built a real-time, multi-level XAI interface for gaze-based interaction using a deep learning model, and evaluated it during a visual search task in XR. A between-subjects study revealed that participants who interacted with XAI made more accurate selections compared to those who did not use the XAI system (i.e., F1 score increase of 10.8%). Additionally, participants who used the XAI system adapted their gaze behavior over time to make more effective selections. These findings suggest that XAI can potentially be used to assist users in more effective collaboration with model-driven interactions in XR.