Eyes on the Game: Deciphering Implicit Human Signals to Infer Human Proficiency, Trust, and Intent

TL;DR

This paper introduces a novel method combining eye gaze and behavioral data to predict human proficiency, trust, and intent in collaborative tasks, demonstrating that integrated models outperform individual data sources.

Contribution

The study presents a new approach to fuse eye gaze and behavioral data for better prediction of human mental states in collaborative environments.

Findings

Integrated models outperform individual data sources.

Eye gaze and behavioral data have complementary predictive strengths.

Aggregation methods can reduce the predictive power of eye gaze.

Abstract

Effective collaboration between humans and AIs hinges on transparent communication and alignment of mental models. However, explicit, verbal communication is not always feasible. Under such circumstances, human-human teams often depend on implicit, nonverbal cues to glean important information about their teammates such as intent and expertise, thereby bolstering team alignment and adaptability. Among these implicit cues, two of the most salient and fundamental are a human's actions in the environment and their visual attention. In this paper, we present a novel method to combine eye gaze data and behavioral data, and evaluate their respective predictive power for human proficiency, trust, and intent. We first collect a dataset of paired eye gaze and gameplay data in the fast-paced collaborative "Overcooked" environment. We then train models on this dataset to compare how the predictive powers differ between gaze data, gameplay data, and their combination. We additionally compare our method to prior works that aggregate eye gaze data and demonstrate how these aggregation methods can substantially reduce the predictive ability of eye gaze. Our results indicate that, while eye gaze data and gameplay data excel in different situations, a model that integrates both types consistently outperforms all baselines. This work paves the way for developing intuitive and responsive agents that can efficiently adapt to new teammates.