Look Hear: Gaze Prediction for Speech-directed Human Attention

TL;DR

This paper introduces ART, a transformer-based model that predicts human gaze patterns during speech-guided object referral tasks, advancing understanding of visual attention in multimodal interactions.

Contribution

The paper presents a novel transformer model and a large-scale gaze dataset for incremental gaze prediction in speech-driven visual tasks.

Findings

ART outperforms existing gaze prediction methods.

The model captures human attention patterns like waiting and scanning.

Created the RefCOCO-Gaze dataset with nearly 20,000 scanpaths.

Abstract

For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is seeing an image and hearing a referring expression defining the object in the scene that should be fixated by gaze. To predict the gaze scanpaths in this incremental object referral task, we developed the Attention in Referral Transformer model or ART, which predicts the human fixations spurred by each word in a referring expression. ART uses a multimodal transformer encoder to jointly learn gaze behavior and its underlying grounding tasks, and an autoregressive transformer decoder to predict, for each word, a variable number of fixations based on fixation history. To train ART, we created RefCOCO-Gaze, a large-scale dataset of 19,738 human gaze scanpaths, corresponding to 2,094 unique image-expression pairs, from 220 participants performing our referral task. In our quantitative and qualitative analyses, ART not only outperforms existing methods in scanpath prediction, but also appears to capture several human attention patterns, such as waiting, scanning, and verification.