Eyes on the Game: Deciphering Implicit Human Signals to Infer Human Proficiency, Trust, and Intent
Nikhil Hulle, St\'ephane Aroca-Ouellette, Anthony J. Ries, Jake, Brawer, Katharina von der Wense, Alessandro Roncone

TL;DR
This paper introduces a novel method combining eye gaze and behavioral data to predict human proficiency, trust, and intent in collaborative tasks, demonstrating that integrated models outperform individual data sources.
Contribution
The study presents a new approach to fuse eye gaze and behavioral data for better prediction of human mental states in collaborative environments.
Findings
Integrated models outperform individual data sources.
Eye gaze and behavioral data have complementary predictive strengths.
Aggregation methods can reduce the predictive power of eye gaze.
Abstract
Effective collaboration between humans and AIs hinges on transparent communication and alignment of mental models. However, explicit, verbal communication is not always feasible. Under such circumstances, human-human teams often depend on implicit, nonverbal cues to glean important information about their teammates such as intent and expertise, thereby bolstering team alignment and adaptability. Among these implicit cues, two of the most salient and fundamental are a human's actions in the environment and their visual attention. In this paper, we present a novel method to combine eye gaze data and behavioral data, and evaluate their respective predictive power for human proficiency, trust, and intent. We first collect a dataset of paired eye gaze and gameplay data in the fast-paced collaborative "Overcooked" environment. We then train models on this dataset to compare how the predictive…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsChild and Animal Learning Development
